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Executive Summary 


The Personnel Development Program to Improve Services and Results for Children with 
Disabilities, also known as the Personnel Development Program (PDP), provides training grants 
to institutions of higher education (IHEs) to support their efforts to directly prepare teachers, 
related services personnel, and leadership personnel to work with children with disabilities. The 
PDP also funds grants to National Centers that support the preparation of special education and 
related personnel through the development and dissemination of evidence-based products and 
services. The PDP is authorized under Section 662 of the Individuals with Disabilities Education 
Act (IDEA) and administered by the U.S. Department of Education’s (ED’s) Office of Special 
Education Programs (OSEP). Between fiscal year (FY) 2005 and FY 2012, appropriations for the 
PDP were between $88 and $91 million per year (figure ES.l). 


Figure ES.l: Annual appropriations for the Personnel Development Program: FY 2005 
through FY 2012 (in millions of current US dollars) 



FIGURE READS: During FY 2005, $90.6 million was appropriated for the PDP. 

NOTE: Current dollars are provided, unadjusted for inflation. 

SOURCE: U.S. Department of Education Congressional Actions Appropriations tables for FY 2005 through FY 2012, 
accessed via http://www2.ed.gov/about/overview/budget/news.html, last modified 02/13/2012. 


The U.S. Department of Education’s Institute of Education Sciences (IES) is authorized by 
Section 664 of IDEA to conduct evaluations of the implementation and impact of programs 
supported under IDEA. In 2007, IES awarded a contract to Westat to conduct a descriptive 
evaluation of the PDP through two distinct substudies — a substudy of PDP National Centers and 
a substudy of PDP training grant applicants. To allow sufficient time for PDP grantees to 
develop products and services that could be described by the evaluation study team and 
evaluated through expert panel review of documents and other materials, the substudies focused 
on grants supported in FY 2006 or FY 2007. Across those 2 years, OSEP disbursed $173.5 


l 



million of PDP funds, of which 93 percent was for new or continuing training grants and 7 
percent was for new or continuing grants to National Centers (figure ES.2). 


Figure ES.2: Disbursement of PDP funding, by type of grant: FY 2006 and FY 2007 (in 
millions of current US dollars) 


FY 2006 
$85.8 


FY 2007 
$87.7 


$ 2.6 $3 - 5 



■ New training 
grants 

■ Continuing 
training grants 

■ New center grants 


Continuing center 
grants 


$ 2.1 $3 - 9 



FIGURE READS: During FY 2006, OSEP disbursed a total of $85.8 million in PDP grants. $18.2 million was disbursed to 
new training grants, $61.4 million to continuing training grants, $2.6 million to new center grants, and $3.5 million to 
continuing center grants. 

NOTE: Calculations are based on actual funding disbursed for (1) new training grants awarded in FY 2006 and FY 2007, 

(2) new center grants awarded in FY 2006 and FY 2007, and (3) continuing center grants awarded in FY 2002 through FY 
2005. Calculations for continuing training grants awarded in FY 2002 through FY 2005 are based on projected funding 
based on original awards. Not included in the figure is an additional $500,000 that was disbursed by the PDP in new grants 
in FY 2007 under the category “Unsolicited.” 

SOURCE: ED Grant Award Database, OSEP Discretionary Grants database, Federal Register Online via U.S. Government 
Printing Office Access. 


This report from the evaluation addresses the following questions regarding PDP National 
Centers funded between fiscal years 2001 and 2007: 

1 . What products were developed and services provided by the PDP National Centers funded 
between FY 2001 and FY 2007, and at what cost? 

2. What were the quality and relevance/usefulness of documented materials and technical 
assistance (TA) provided by PDP National Centers funded between FY 2001 and FY 
2007? 

The substudy of PDP training grant applicants focused on courses of study that were at the center 
of applications. A “course of study” was defined as a set of postsecondary courses in a 
particular campus or online setting to prepare candidates to perform a particular professional 
or paraprofessional role, and almost always resulting in a degree, a particular credential 


u 


(license, certificate, or endorsement), or both. The report addresses the following questions 
regarding PDP training grant applicants from the FY 2006 and FY 2007 competitions: 

1 . What were the characteristics of funded courses of study at IHEs awarded PDP training 
grants in FY 2006 or FY 2007? 

2. How did funded courses of study use PDP training grant funding? 

3. How many scholars 1 enrolled in the funded courses of study, completed their programs, or 
dropped out before completion? 

4. What were the quality and relevance/usefulness of new or significantly modified 
components for funded courses of study? 

5. What became of courses of study that did not receive PDP training grant funding? 

The sections below provide, for each substudy, background on the data collection and the 
analysis conducted to answer the research questions and a summary of key findings. 


Substudy of PDP National Centers 

The substudy of the PDP National Centers focused on centers that were actively funded, or in 
no-cost extension status, at the time that the study began in 2007. Twelve of the 14 centers 
funded in FY 2001 through FY 2007 were included in the in-depth analysis for the substudy of 
PDP National Centers (table ES.l). 2 For each of these centers, the evaluation study team 
collected data through (1) a review of relevant extant data; (2) a standardized inventory of center 
products and services; (3) a semi-structured interview with the center director; and (4) expert 
panel review of a sample of up to 10 products and services per center, including up to 3 signature 
works. 3 A product or service was defined as a single activity or group of closely related activities 
designed to achieve a specific outcome for a specific audience. The evaluation study team 
sampled products and services from all those that were deemed feasible for review through the 


1 Scholar means an individual who is pursuing a degree, license, endorsement, or certification related to special education, related 
services, or early intervention services. 

2 Of the two National Centers not sampled for in-depth analysis, one was funded for 2007-12 to focus on autism, but had fewer 
products available for review in 2010 than an earlier Autism Center funded for 2003-07, which was included in the sample. The 
other National Center not sampled for in-depth analysis was funded for 2002-06 to focus on large-scale special education 
research. This center was funded at the same university and with the same project director as a National Center funded for 
2006-201 1 to focus on the analysis of large-scale databases, and the later center was included in the sample. 

3 Signature works were those that, in the judgment of each center’s director, (1) were prominent in tenns of the proportion of the 
center’s overall effort, (2) were exclusively or nearly exclusively a product of the center’s effort, (3) were predominately 
supported by PDP grant funds or unlikely to have occurred without PDP grant support, and (4) made a consequential 
contribution to the field or to the class of customers served. 


iii 



examination of documents or other tangible materials such as videos. 4 Panelists used center- 
specific rubrics developed by the evaluation study team. Data collection occurred between fall 
2008 and summer 2010, which was before the end of the PDP grant for six of the National 
Centers studied. 

Table ES.l: Overview of PDP National Centers funded between FY 2001 and FY 2007 


National Center 

Grantee 

Funding period 
(fiscal years) 

Funding 

amount 

The IDEA and Research for Inclusive Settings 
Center for Faculty Enhancements 

Vanderbilt University 

2001-06 

$4,249,995 

Carolina Interdisciplinary Large-Scale Policy 
Research Training Center 

University ofNorth Carolina 
at Chapel Hill 

2002-06 

$1,649,846 

Policy > Leadership Training in Conducting 
Large-Scale Special Education Research 

University of Maryland 

2002-06 

$1,750,000 

Center for Improving Teacher Quality 

Council of Chief State 
School Officers 

2002-07 

$4,983,285 

Center for Early Childhood Education/Early 
Intervention Personnel Preparation 

University of Connecticut 

2003-07 

$2,961,124 

Professional Development in Autism Center 

University of Washington 

2003-07 

$5,049,993 

National Center for Leadership in Visual 
Impairments 

Pennsylvania College of 
Optometry, later transferred 
to Salus University 

2004-09 

$5,054,671 

Interdisciplinary Training in Analysis of 
Large-Scale Databases Center 

University of Maryland 

2006-11 

$2,499,576 

IDEA and Research for Inclusive Settings 
Center for Training Enhancements 

Vanderbilt University 

2006-11 

$6,750,000 

National Institute for Urban School 
Improvement Principal Leadership Academies 
Initiative 

Arizona State University, 
later transferred to University 
of Arizona 

2006-11 

$1,424,999 

National Professional Development Center on 
Inclusion 

University ofNorth Carolina 
at Chapel Hill 

2006-11 

$2,372,165 

The Center to Mobilize Early Childhood 
Knowledge 

University ofNorth Carolina 
at Chapel Hill 

2007-12 

$2,998,870 

National Center to Inform Policy and Practice 
in Special Education Professional 
Development 

University of Florida 

2007-12 

$2,500,000 

National Professional Development Center on 
Autism Spectrum Disorders 

University ofNorth Carolina 
at Chapel Hill 

2007-12 

$5,000,000 


TABLE READS: The IDEA and Research for Inclusive Settings Center for Faculty Enhancement (IRIS-I), based on a PDP grant 
awarded to Vanderbilt University, was funded from FY 2001 to FY 2006 at a total amount of $4,249,995. 

NOTE: The two PDP National Centers listed in this table in italics were funded between FY 2001 and FY 2007, but were not included in 
in-depth data collection for the evaluation because of their similarity to two other National Centers. 

SOURCE: Center grant applications provided by the Office of Special Education Programs (OSEP), U.S. Department of Education, and 
the Online Discretionary Grants Public Database provided through the OSEP Technical Assistance and Dissemination Network. 


4 Reasons products were not reviewed by expert panels included the following: missing or insufficient documentation in the 
judgment of the evaluation study team (for example, a PowerPoint presentation only); lack of stability of documentation (as in 
the case of a website as the only documentation); a paper for which a National Center staff member was not the first author 
listed; submission of promotional materials as documentation; submission of internal planning documents as documentation; or 
the bundling of a product with another product, making a distinct review infeasible. 




Findings from the Substudy of PDP National Centers 

PDP National Centers are organized around a variety of priorities, including providing support to 
states, districts, and schools; facilitating the preparation of practitioners and researchers; and 
enhancing evidence-based practice around specific disability areas such as autism and visual 
impairment. Key findings, with additional detail, are presented below by research question. 

What products were developed and services provided by the PDP National Centers funded 
between FY 2001 and FY 2007, and at what cost? Across 12 National Centers funded by the 
PDP between FY 2001 and FY 2007, project directors identified 759 products or services 
developed by the time of data collection (fall 2008 to summer 2010) (table ES.2). The greatest 
number of products and services identified were presentations and webinars (356). Ten centers 
were able to report costs for product development and dissemination, and they reported costs for 
386 products/services (51 percent). Those costs were $19.5 million. The evaluation study team 
grouped products and services reported by center directors into 10 categories. The largest share 
of costs reported for product development and dissemination was for the provision of 69 
conferences, institutes, or workshops (48 percent). These included center-produced events for the 
purposes of training recipients, providing general TA, and disseminating information to targeted 
recipients. 

What were the quality and relevance/usefulness of documented materials and TA provided by 
PDP National Centers funded between FY 2001 and FY 2007? Of the 101 products/services 
selected across the centers for review by panels of technical and content experts, 1 5 were 
identified by center staff as signature works. Each product was rated in the summer of 2010 by 
three trained panelists 5 using center-specific rubrics containing a series of indicators grouped 
into two dimensions — quality and relevance/usefulness. Rating values ranged from 1 (very low) 
to 5 (very high). For the 15 reviewed products identified by center staff as signature works, the 
mean rating was 4.13 for quality and 4.25 for relevance/usefulness. Seventy-seven percent of the 
quality ratings and 82 percent of the relevance/usefulness ratings for the signature products were 
high or very high (figure ESA). Across the 86 nonsignature products, the mean rating was 4.11 
for quality and 3.91 for relevance/usefulness. Seventy-three percent of quality ratings and 68 
percent of relevance/usefulness ratings for the nonsignature products were high or very high. 
These ratings do not generalize to center products or services that lacked sufficient 
documentation for review, such as products represented only by websites or PowerPoint 
presentations. 


5 


One panelist was exempt from rating one product due to a predetermined conflict of interest. 
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Table ES.2: PDP National Center products and services, with reported costs and percentage of 

reported costs, by type of product 




Number of 






products 


Average 

Percent of 


Number of 

with cost 

Reported 

cost per 

reported 

Product type 

products 1 

information 

costs 2 

product 

costs 

All products 

759 

386 

$19,509,083 

$50,542 

100 

Briefs or short reports 

71 

47 

301,246 

6,409 

2 

Conferences, institutes, or 
workshops 3 

88 

69 

9,341,040 

135,377 

48 

Intensive, focused technical 

Q 

Q 

825,240 

91,693 

A 

assistance (TA) 4 

y 

y 


Materials supporting universal 
TA 5 

li 

li 

1,357,937 

123,449 

7 

Presentations or webinars 6 

356 

151 

619,730 

4,104 

3 

Research papers, articles, 
dissertations, or book chapters 

147 

27 

230,975 

8,555 

1 

University courses, programs, 
or training models 

5 

4 

1,100,770 

275,193 

6 

Web-based training modules, 
including supporting materials 

56 

56 

3,435,091 

61,341 

18 

Websites, including activities 
facilitating website use 

8 

6 

1,852,371 

308,729 

9 

Other products or services 7 

8 

6 

444,683 

74,114 

2 


TABLE READS: Across the PDP National Centers, a total of 759 products had been developed by the time of data collection, 

with a reported total cost of $19.5 million and average cost of $50,542 for 386 of these products. 

1 Six centers were still active at the time of data collection. Counts of products and services in this table include only those 
developed by the time of data collection. 

2 Ten of the 12 centers were able to report costs for some or all of their products/services. Costs associated with monetary 
support for scholars are not included in this table. 

3 Included center-produced events for the purposes of training recipients, providing general TA, or disseminating information to 
targeted recipients. 

4 Involved an individualized, ongoing relationship between the center and the TA recipient. 

5 Included products developed for individuals’ independent use through their own initiative. Does not include products counted 
in other categories that may also serve TA purposes, such as briefs, institutes, presentations, etc. 

6 Included individual, invited presentations by center staff or center-affiliated scholars, as well as individual, center-produced 
events for information dissemination. 

7 Products and services categorized as “other” included promotional and informational materials about center products/services, 
as well as outreach, recruitment, and networking activities. 

SOURCE: Inventory of products and services collected from two center directors in September 2008 and January 2009 and 

remaining center directors November 2009 through March 2010. 
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Figure ES.3: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature 
products/services of 12 PDP National Centers 


Percent 


1 1 1 


Very Low 
Low 

■ Moderate 

■ High 


■ Very High 


Quality Relevance/ Quality Relevance/ 

Usefulness Usefulness 

Signature products/services 1 Nonsignature products/services 2 

(n=15 products/services; (n=86 products/services for quality, 

mean of 4.13 for 247 quality ratings n=85 for relevance/usefulness; 

and 4.25 for 266 relevance/usefulness ratings) mean ratings of 4.11 for 1,452 quality ratings 

and 3.91 for 1,071 relevance/usefulness ratings) 

FIGURE READS: For the 15 signature products/services reviewed by an expert panel, 1 percent of the 247 quality ratings were 
at the level of 1 (very low), 1 percent were 2 (low), 20 percent were 3 (moderate), 39 percent were 4 (high), and 38 percent 
were 5 (very high). The mean quality rating was 4.13. 

'Center staff were asked to identify up to 10 percent of products/services as signature, defined as works that (1) were prominent 
in terms of the proportion of the center’s overall effort, (2) were exclusively or nearly exclusively a product of the center’s 
effort, (3) were predominately supported by PDP funds or unlikely to have occurred without PDP support, and (4) made a 
consequential contribution to the field or customers served. Two centers did not identify any of their products/services as 
signature works and are therefore not included in the distribution of ratings for signature products/services. 

2 One nonsignature product was not included in analyses for relevance/usefulness because it received NAs for 13 out of 18 
indicator-level ratings. One reviewer was exempt from rating one nonsignature product due to a predetermined conflict of 
interest; those data were treated as missing and are not included in the analysis. 

SOURCE: Expert panel reviews conducted May through September 2010. 



Substudy of PDP Training Grant Applicants 

IHEs applied to a particular PDP priority/focus area based on the area of preparation upon which 
their course of study was or was expected to be focused, including the degree level for which 
candidates would be prepared. OSEP grouped priority/focus areas for the direct preparation of 
personnel into two overarching priorities for the FY 2006 training grant competitions: 

(1) a Combined priority with five focus area competitions — two competitions focused on 
training staff based on the severity of disability to be served (Low Incidence, High 
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Incidence), two focused on the professional or paraprofessional role (Early Intervention 
and Early Childhood, Related Services), and one focused on the characteristics of the 
institution applying for funding (Minority Institutions), and 

(2) a Leadership priority primarily focused on training faculty in special education and 
related service areas but also including grants to support the training of administrators. 

In FY 2007, the Leadership and Combined priority areas remained, although the High-Incidence 
focus area was removed from Combined, and a new priority area for “Special Education 
Preservice Training Improvement Grants” was created to replace it. The new priority focused on 
improvement of preservice training programs themselves and, unlike all the other priority/focus 
areas, prohibited rather than required scholar stipends. 

IHEs could submit multiple applications in a given year within or across priority/focus areas for 
funding related to different courses of study, and a single PDP application could include one or 
more courses of study. The substudy of PDP training grant applicants focused on the 
predominant course of study, defined, for funded applications, as the course of study that 
received the largest amount of grant funds in the 2008-09 academic year (AY) and, for 
nonfunded applications, as the course of study for which the most funding was requested at the 
time of application. The initial study sample consisted of 537 applications: 190 that were 
funded by the PDP in FY 2006 or FY 2007, and 347 that were not funded. 

Data on PDP training grant applicants were collected through (1) a review of relevant extant 
data, (2) a web-based survey of funded and nonfunded applicants, and (3) expert panel review of 
the quality and relevance/usefulness of a sample of new or significantly modified components 
within funded courses of study. The web-based survey was administered from the fall of 2009 
through the winter of 2010, and the expert panel review of course of study components occurred 
in the summer of 2010, focusing on materials collected from grantees in the spring and early 
summer of 2010. 

The panels to review PDP-funded course of study components consisted of 14 academic experts 
with relevant content knowledge. Trained panelists rated sampled course of study components on 
a series of indicators grouped into two dimensions — quality and relevance/usefulness — with 
rating values ranging from 1 (very low) to 5 (very high). The mean and distribution of ratings 
were analyzed separately by dimension. Ratings of quality and relevance/usefulness only reflect 
sampled components for which review was feasible; they do not represent the quality or 
relevance/usefulness of each course of study as a whole. 

Findings from the Substudy of PDP Training Grant Applicants 

Of the 537 applications from IHEs to the FY 2006 and FY 2007 PDP training grant 
competitions, 190 (35 percent) were funded. The percentage of successful applications was 
higher for public IHEs than for private IHEs, for doctorate-granting IHEs than non-doctorate- 
granting IHEs, and for minority institutions than for non-minority institutions (figure ES.4). Key 
findings are presented below by research question. 
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Figure ES.4: Percent of FY 2006 and FY 2007 applications funded, by institution of higher 
education characteristic 

Percent of applications 

50 
45 
40 
35 
30 
25 
20 
15 
10 
5 
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FIGURE READS: Across the FY 2006 and FY 2007 PDP competitions, 37 percent of applications from public IHEs were 
funded compared to 32 percent of applications from private IHEs; 38 percent of applications from doctorate-granting IHEs 
were funded compared to 25 percent from non-doctorate-granting IHEs; and 38 percent of applications from minority 
institutions were funded compared to 34 percent of applications from non-minority institutions. 

NOTE: The initial study sample included 537 applications. Six nonfunded applications were removed from the sample because 
the evaluation study team was unable to identify either the project director or the course of study within the application, resulting 
in a study sample of 531 applications. Public institutions are those whose programs and activities were operated by publicly 
elected or appointed school officials and which were supported primarily by public funds at the time of PDP application; 
doctorate-granting institutions are those that offered the doctor’s degree at the time of PDP application; and minority institutions 
are defined as institutions that, as of 20 1 0, ( 1 ) had a 25 percent or more minority enrollment (excluding the other or nonresident 
categories), (2) had a historical distinction as minority institutions, or (3) were designed to serve special populations. 

SOURCE: IES Integrated Post-secondary Education Data System (IPEDS) database, and the National Outreach and Technical 
Assistance Center on Discretionary Awards for Minority Institutions (Monarch Center). 
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What were the characteristics of funded courses of study at IHEs awarded PDP training 
grants in FY 2006 or FY 2007? Of the 190 applications that were funded in the FY 2006 and 
FY 2007 training grant competitions across all priority areas, 84 percent were from public 
institutions, 88 percent were from doctorate-granting institutions, and 48 percent were from 
minority institutions. Eleven percent of funded applications contained multiple courses of study. 
Across all priority/focus areas, 53 percent of funded courses of study differed substantially from 
any course of study at the submitting IHE at the time of PDP application. The remaining 47 
percent were existing courses of study that were supported or expanded through the PDP grant. 

Overall, 45 percent of the funded courses of study proposed to award master’s or educational 
specialist degrees. Forty-three percent of funded courses of study proposed to prepare special 
education teachers; 24 percent proposed to prepare leadership personnel; and the remainder 
proposed to prepare other personnel such as early childhood specialists, speech-language 
pathologists, or psychologists. 

Regarding the category of disabilities personnel would serve, 37 percent of the funded courses of 
study proposed to prepare personnel to serve all categories of disability, while a combined 34 
percent proposed to prepare personnel to serve particular high- or low-incidence disability 
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categories. Additionally, 48 percent of funded courses of study proposed to prepare personnel to 
serve general school-age children, while 22 percent proposed to prepare personnel to serve birth 
to preschool, and another 22 percent to prepare personnel to serve birth to 2 1 or an unspecified 
age. 

How did funded courses of study use PDP training grant funding? Across priority/focus areas 
during the 2008-09 academic year, where scholar stipends were required, an average of 73 
percent of grant dollars went to monetary support for scholars, and 18 percent supported faculty 
(table ES.3). Among Preservice Improvement grant recipients, which were prohibited from 
spending PDP grant funds for scholar stipends, an average of 6 1 percent of grant dollars were 
spent on faculty during academic year 2008-09. For the 159 PDP-supported predominant courses 
of study in priority/focus areas where scholar stipends were required, 88 percent of enrolled 
scholars received monetary support from the PDP grant in academic year 2008-09, and the mean 
amount of that support was $1 1,558. 


Table ES.3: Funding allocations per predominant course of study funded in FY 2006 and 
FY 2007: Academic year 2008-09 


Mean total amount of 

Mean percent of grant dollars spent on 


grant dollars spent during 
AY 2008-09 

Curriculum 

Faculty 

Scholars 1 

Other 

All priority/focus areas with 
required scholar stipends 
(n=159) 

$157,016 

2 

18 

73 

7 

Preservice Improvement 
(High Incidence FY 2007) 
(n=21) 

$77,197 

14 

61 

0 

25 


TABLE READS: Among 159 FY 2006 and FY 2007 funded predominant courses of study where scholar stipends were 
required, the mean total amount of grant funds spent during AY 2008-09 was $157,016. Of these funds, a mean of 2 percent 
was spent on curriculum, 18 percent on faculty, 73 percent on scholars, and 7 percent on other expenses. 

’The Preservice Improvement (High Incidence FY 2007) priority/focus area did not permit grant funds to be spent on 
scholar stipends. All other areas required that some grant funds be spent on scholar stipends. 

NOTE: One funded course of study in the Preservice Improvement (High Incidence FY 2007) priority area had not yet been 
developed by AY 2008-09. Five funded course of study did not complete survey item 22. Data from these courses of study 
are therefore not included in analyses for this table. Detail may not sum to totals due to rounding. 

SOURCE: IHE Course of Study Survey - Item 22. 


How many scholars enrolled in the funded courses of study, completed their programs, or 
dropped out before completion? Across all courses of study funded in the FY 2006 or FY 2007 
competitions, between the year of grant funding and AY 2008-09, 3,038 scholars completed his 
or her course of study, averaging slightly more than 17 scholars per course of study. Eighty-six 
percent of course of study completers earned state-issued credentials, and 46 percent earned a 
master’s or education specialist degree. The mean number per funded course of study of 
enrollees, new scholars, and degree and credential earners (i.e., completers) increased from the 
academic year prior to funding to the first academic year of full grant funding. In AY 2008-09, 
174 of 4,199 enrolled scholars dropped out. Thirty-eight percent of funded courses of study had 
at least one scholar leave. 
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What were the quality and relevance/usefulness of new or significantly modified components 
for funded courses of study? Respondents to the course of study survey identified changes made 
to their courses of study since the time of their application. Possible changes included the 
addition, significant modification, or discontinuation of course of study components, including 
changes to courses, training units or modules (including field experience), recruitment plans, 
mentoring programs, faculty positions, or the overall organization of a course of study. Using the 
survey data and additional documents provided by funded courses of study, the evaluation study 
team sampled 134 new or significantly modified components from 99 different courses of study 
as suitable for expert panel review. Experts rated all 134 components for quality and 70 
components for relevance/usefulness. 6 Fifty-eight percent of the experts’ ratings of quality, and 
57 percent of the experts’ ratings of relevance/usefulness, were “high” or “very high” (figure 
ES.5). Sixteen percent of quality ratings, and 21 percent of relevance/usefulness ratings, were 
“low” or “very low.” The mean rating for quality (3.7 1 on a 5-point scale, with 5 being the 
highest) was between “moderate” and “high,” as was the mean rating for relevance/usefulness 
(3.63). 

What became of courses of study that did not receive PDP training grant funding? Thirty-four 
percent of courses of study that were not funded through FY 2006 and FY 2007 competitions 
were developed or maintained without PDP funding. Among proposed nonfunded courses of 
study that were intended to be substantially different from any existing course of study at the 
IHE, 8 percent were developed without PDP funding. Of the nonfunded preexisting courses of 
study that were intended to be maintained or enhanced through a PDP grant, 88 percent were 
maintained without PDP funding. 

Of IHEs with PDP training grant applications that were not funded in FY 2006, 70 percent 
reapplied to the same priority/focus area within 4 years, and 50 percent were funded. Of IHEs 
with nonfunded grant applications in FY 2007, 57 percent reapplied to the same priority/focus 
area within 3 years, and 33 percent were funded. 


6 The rubrics developed to evaluate each type of course of study component (appendix E) originally distingused some indicators 
as measures of quality and other indicators as measures of relevance/usefulness. However, during testing of the rubrics, expert 
consultants recommended that it would be more appropriate to classify all of the indicators for components involving new 
faculty, fieldwork, and mentoring programs as quality indicators. 
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Figure ES.5. Expert panelist ratings (on a 5-point scale, with 5 being the highest) of 

quality and relevance/usefulness of new or significantly modified course of 
study components funded with FY 2006 or FY 2007 PDP training grants 
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(n=134 course of study components; (n=70 course of study components; 

mean of 3.71 for 2,604 quality ratings) mean of 3.63 for 630 

relevance/usefulness ratings) 


FIGURE READS: For the 134 new or significantly modified course of study components reviewed by an expert panel for 
quality, 5 percent of the 2,604 ratings were at the level of 1 (very low), 1 1 percent were 2 (low), 26 percent were 3 (moderate), 25 
percent were 4 (high), and 33 percent were 5 (very high). The mean quality rating was 3.71. 

NOTE: Data in this table were weighted using the inverse of the probability of selection for each sampled change component to 
allow generalization to all change components reported by grantees. Relevance/usefulness composites are based on ratings from 
Courses or Classes and Training Units or Modules components only. 

SOURCE: Expert panel reviews, conducted July through October 2010. 
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1. Introduction 


The Personnel Development Program to Improve Services and Results for Children with 
Disabilities is authorized under Section 662 of the Individuals with Disabilities Education Act 
(IDEA) and is known as the Personnel Development Program (PDP). The PDP is administered 
by the U.S. Department of Education’s (ED’s) Office of Special Education Programs (OSEP) in 
the Office of Special Education and Rehabilitative Services. The program’s purposes are (1) to 
help address state-identified needs for highly qualified personnel in special education, related 
services, early intervention, and regular education to work with children with disabilities and 
(2) to ensure that those personnel have the skills and knowledge needed to serve these children. 
The PDP provides training grants to institutions of higher education (IHEs) to support their 
efforts to directly prepare teachers, related services personnel, and leadership personnel. The 
PDP also funds a series of National Centers that indirectly support the preparation of personnel 
through capacity-building and the development of evidence-based products and services. 

ED’s Institute of Education Sciences (IES) is authorized by Section 664 of IDEA to 
conduct evaluations of the implementation and impact of programs supported under IDEA. In 
2007, IES awarded a contract to Westat to conduct a descriptive evaluation of the PDP through 
two distinct substudies — a substudy of PDP National Centers and a substudy of PDP training 
grant applicants. This chapter provides background on the program. Chapter 2 of this report 
describes the design for the substudy of PDP National Centers, and chapters 3 and 4 present the 
corresponding findings overall and by center. Chapter 5 presents the design for the substudy of 
PDP training grant applicants, and chapter 6 presents the corresponding findings. 

History and Structure of the Personnel Development Program 

Federal funding for the preparation of special education personnel began in 1958 with 
passage of Public Law (P.L.) 85-926, the Education of Mentally Retarded Children Act. In 1975, 
P.L. 94-142 — the Education of All Handicapped Children Act (commonly known as EHA) — 
called for a free appropriate public education for all children with disabilities ages 3 through 2 1 . 
EHA required grant applicants to OSEP for special education personnel preparation grants to 
demonstrate that their proposals would meet needs identified in their respective state’s 3 -year 
plans for comprehensive systems of personnel development (Fiore et al. 2001). With the 
establishment of ED in 1979 by the Department of Education Organization Act (P.L. 96-88), the 
Personnel Preparation Program (PPP) was administered by the Division of Personnel Preparation 
within OSEP (Campeau, Appleby, and Stoddart 1987). Reauthorization of EHA in 1990 resulted 
in the newly named IDEA. 

When IDEA was reauthorized in 1997 (P.L. 105-17), the PPP became focused on grants 
for leadership personnel and personnel preparation to educate and support children with low- 
incidence or high-incidence disabilities. Grant applicants were still required to address state- 
identified needs. The 1997 reauthorization of IDEA included a new requirement that students 
funded through PPP grants work in that field for 2 years for every year of support they received. 
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The 2004 reauthorization of IDEA (P.L. 108-446) renamed the PPP as the Personnel 
Development Program (PDP). Priority areas for the direct preparation of personnel were 
organized into two priorities for fiscal year (FY) 2005 training grant competitions: (1) a 
Leadership priority area, and (2) a Combined priority area for training personnel with a focus on 
Low-Incidence disabilities, High-Incidence disabilities, Early Intervention and Early Childhood 
programs, Related Services, and Minority Institutions. In FY 2007, the High-Incidence focus 
area was removed from Combined and replaced with a new priority area for Special Education 
Preservice Training Improvement Grants. The new priority placed an emphasis on “highly 
qualified teacher requirements,” focused on improvement of preservice training programs 
themselves, and, unlike the other priorities, prohibited rather than required scholar stipends. 

In addition to training grants based on priorities for direct personnel preparation, there 
have been one-time or recurring PPP/PDP grants to provide indirect support for personnel 
preparation. The development of these National Centers was motivated by a need for research, 
technical assistance, or personnel preparation resources. Centers have provided a variety of 
national capacity-building products and services to a variety of audiences, such as current and 
future researchers; trainers and consultants in various disability fields; and special education 
leaders at the national, state, and local levels. 

When federal funding for personnel preparation began in 1958, $1 million in current year 
dollars were authorized. Between FY 2005 and FY 2012, appropriations for the PDP have been 
between $88 and $91 million each year; FY 2012 funding was $88.3 million (figure 1.1). 


Figure 1.1: Annual appropriations for the Personnel Development Program: FY 2005 
through FY 2012 (in millions of current US dollars) 


Appropriations in millions of US $ 



FIGURE READS: During FY 2005, $90.6 million was appropriated for the PDP. 

NOTE: Current dollars are provided, unadjusted for inflation. 

SOURCE: U. S. Department of Education Congressional Actions Appropriations tables for FY 2005 through FY 2012, 
accessed via http://www2.ed.gov/about/overview/budget/news.html, last modified 02/13/2012. 
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The substudies of PDP National Centers and PDP training grant applicants focused on 
grants supported in FY 2006 or FY 2007. This timeframe allowed sufficient time for PDP 
grantees to develop products and services that could be described by the evaluation study team 
and evaluated through expert panel review of documents and other materials. Across these fiscal 
years, OSEP disbursed $173.5 million (figure 1.2). Of this total, 93 percent ($161.3 million) was 
disbursed in training grants, and 7 percent ($12.2 million) was disbursed for National Center 
grants, including both new grants and continuing grants. 

Figure 1.2: Disbursement of PDP funding, by type of grant: FY 2006 and FY 2007 (in 
millions of current US dollars) 


FY 2006 
$85.8 


FY 2007 
$87.7 


$ 2.6 $3 - 5 



■ New training 
grants 

■ Continuing 
training grants 

■ New center grants 


Continuing center 
grants 


$ 2.1 $3 - 9 



FIGURE READS: During FY 2006, OSEP disbursed a total of $85.8 million in PDP grants. $18.2 million was disbursed to 
new training grants, $61.4 million to continuing training grants, $2.6 million to new center grants, and $3.5 million to 
continuing center grants. 

NOTE: Calculations are based on actual funding disbursed for (1) new training grants awarded in FY 2006 and FY 2007, 

(2) new center grants awarded in FY 2006 and FY 2007, and (3) continuing center grants awarded in FY 2002 through FY 
2005. Calculations for continuing training grants awarded in FY 2002 through FY 2005 are based on projected funding 
based on original awards. Not included in the figure is an additional $0,500 million that was disbursed by the PDP in new 
grants in FY 2007 under the category “Unsolicited.” 

SOURCE: ED Grant Award Database, OSEP Discretionary Grants database, Federal Register Online via U.S. Government 
Printing Office Access. 


During the FY 2006 and FY 2007 competitions, OSEP reviewed 269 training grant 
applications each year. Thirty percent of reviewed applications were awarded new grants in FY 
2006, and 41 percent were awarded new grants in FY 2007 (table 1.1). 


3 


Table 1.1: PDP training grant applications and awards, by PDP priority/focus area: 

FY 2006 and FY 2007 



FY 2006 

FY 2007 

Number of PDP training grant applications reviewed 



All priority/focus areas 

269 

269 

Leadership 

84 

75 

Combined 

185 

158 

Early Intervention/early Childhood 

21 

26 

Low Incidence 

25 

44 

High Incidence 

58 

t 

Related Services 

44 

33 

Minority Institutions 

37 

55 

Preservice Improvement (High Incidence) 

t 

36 

Percentage of applications awarded new grants 



All priority/focus areas 

30% 

41% 

Leadership 

26 

32 

Combined 

32 

40 

Early Intervention/ Early Childhood 

52 

58 

Low Incidence 

52 

41 

High Incidence 

22 

t 

Related Services 

25 

45 

Minority Institutions 

30 

27 

Preservice Improvement (High Incidence) 

t 

61 


TABLE READS: Of the 269 applications for PDP training grants reviewed in the FY 2006 competition, 30 percent were 
awarded new grants. 

' Priority/focus area is not applicable for the given fiscal year. 

SOURCE: OSEP PDP application slates for FY 2006 and FY 2007. 


In academic year (AY) 2005-06, 338 personnel preparation programs existed nationally 
that had certificates/degrees and focus areas of relevance to the PDP Leadership priority area 
(table 1.2). Approximately 25 percent of these training programs applied to the FY 2006 PDP 
Leadership competition, and approximately 7 percent received PDP awards. For the 
Combined/Preservice Improvement competition, approximately 5 percent of 3,914 relevant 
programs applied and approximately 2 percent were funded. 

In addition to grants for the direct preparation of personnel, OSEP awarded 14 grants for 
the development of PDP National Centers between FY 2001 and FY 2007 (table 1.3). Total 
funding to be awarded for these centers throughout their funding cycles was $50.2 million. 
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Table 1.2: Personnel preparation programs in the U.S. relevant to PDP priority areas and 
the representative percentage of programs that applied for and received PDP 
grants, by PDP competition: FY 2006 and FY 2007 


Relevant PDP competition 

Number of relevant 
existing programs in 
U.S. in the 
corresponding 
academic year 

Percent of relevant 
existing programs 
that applied to the 
relevant PDP 
competition 

Percent of relevant 
existing programs 
that received grants 
through the relevant 
PDP competition 

FY 2006 Leadership 

338 

24.9% 

6.5% 

FY 2006 Combined 

3,770 

4.9 

1.6 

FY 2007 Leadership 

351 

21.4 

6.8 

FY 2007 Combined/ 
Preservice Improvement 

3,914 

5.0 

2.2 


TABLE READS: For the PDP FY 2006 Leadership competition, there were 338 existing personnel preparation programs in 
the U.S. in AY 2005-06 with relevant certificates/degrees and focus areas. The number of programs that applied to the PDP 
FY 2006 Leadership competition represented 24.9 percent of existing programs; the number that received grants represented 
6.5 percent of existing programs. 

NOTE: Personnel preparation programs were coded according to the 2000 taxonomy of the Classification of Instructional 
Programs (CIP), a detailed coding system for postsecondary instructional programs (http://nces.ed.gov/ipeds/about/). 

SOURCE: U.S. Department of Education’s Integrated Post-secondary Education Data System (IPEDS). 


Table 1.3: PDP center grant awards and funding: FY 2001 through FY 2007 


Fiscal year 

Number of new center 
grants awarded 

Total funding to be awarded throughout 
grants (in millions of dollars) 1 

2001-07 

14 

$ 50.2 

2001 

1 

4.2 

2002 

5 

16.4 

2003 

0 

0 

2004 

1 

4.6 

2005 

0 

0 

2006 

4 

14.5 

2007 

3 

10.5 


TABLE READS: During FY 2001 through FY 2007, OSEP funded 14 new PDP National Centers, awarding a total of $50.2 
million in new center grants. 

1 Calculated based on the maximum total funding available for each grant throughout its funding cycle, as opposed to 
incremental funding in any single fiscal year. Current dollars are provided, unadjusted for inflation. 

SOURCE: ED Grant Award Database, OSEP Discretionary Grants database. 
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2. Design for the Substudy of PDP National Centers 


For the substudy of PDP National Centers, the evaluation study team (1) documented the 
products and services generated by 12 National Centers funded by the PDP between FY 2001 
and FY 2007 and the costs of the development and dissemination of those products and services, 
and (2) assessed the quality and relevance/usefulness of a sample of those products and services. 
For each of the 12 centers, data were collected through (1) a review of relevant extant data, 

(2) an inventory of center products and services, (3) an interview with the center director, and 
(4) expert review of a sample of center products and services. Table 2. 1 lists the research 
questions for this study, along with the sources of data and analyses for each question. 


Table 2.1: Research questions, data sources, and analyses for the substudy of PDP 

National Centers 


Research question 

Data sources 

Analyses 

1 . What products were developed 
and services provided by the 
PDP National Centers funded 
between FY 2001 and FY 2007, 
and at what cost? 

Review of extant data, 
inventories of products 
and services, interviews 
with center directors. 

Description of the purpose, 
audience, and funding of each 
center. Categorized list of products 
developed and services provided 
by each center, with associated 
costs. Cross-center summary and 
center-specific analyses. 

2. What were the quality and 
relevance/usefulness of 
documented materials and 
technical assistance (TA) 
provided by PDP National 
Centers funded between FY 
2001 and FY 2007? 

Rubric-based reviews, 
by expert panel, of a 
sample of center 
products and services 
produced with grant 
funding. 

Ratings of quality and 
relevance/usefulness of a sample 
of materials developed and 
services provided, along with 
estimates of inter-rater agreement. 
Means and distributions of ratings. 
Cross-center summary and center- 
specific analyses. 


Selection of Centers 

OSEP funds PDP National Centers to support the preparation of special education and related 
personnel by (1) enhancing the knowledge base through research or the organization of existing 
knowledge for specific audiences, (2) developing evidence-based products and services, and 
(3) providing TA. The substudy of the PDP National Centers focused on centers that were 
actively funded, or in no-cost extension status, at the time that the study began in 2007. Between 
FY 2001 and FY 2007, OSEP funded 14 PDP National Centers. These centers were organized 
around a variety of priorities, including providing support to states, districts, and schools; 
facilitating the preparation of practitioners and researchers; and enhancing evidence-based 
practice around specific disability areas such as autism and visual impairment (table 2.2). 
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Table 2.2: Overview of PDP National Centers funded in FY 2001 through FY 2007 


Center name and purpose 

Grantee 

Intended 

audience 

Funding 

period 

Funding 

amount 

The IDEA and Research for Inclusive 
Settings Center for Faculty Enhancement 
(IRIS-I): Designed to assist college faculty to 
better prepare general education teachers, 
administrators, school counselors, and school 
nurses who have key roles in educating 
students with disabilities through the 
provision of online modules and course 
enhancement materials. 

Vanderbilt 

University 

Institution of 
higher 

education (IHE) 
faculty and staff 

2001-06 

(no-cost 

extension 

through 

7/2007) 

$4,249,995 

Carolina Interdisciplinary Large-Scale 
Policy Research Training Center 
(CILSPRT): Designed to prepare special 
education doctoral students and postdoctoral 
fellows to conduct large-scale and 
longitudinal policy studies under the guidance 
of interdisciplinary faculty and research 
partners. 

University of 
North 
Carolina at 
Chapel Hill, 
Frank Porter 
Graham Child 
Development 
Institute 

Doctoral 
students and 
postdoctoral 
fellows 

2002-06 

(no-cost 

extension 

through 

9/2007) 

$1,649,846 

Policy Leadership Training in Conducting 
Large-Scale Special Education Research: 

Designed to support doctoral students and 
post-doctoral fellows to conduct large-scale 
research in special education through 
interdisciplinary coursework and mentored 
internships. 

University of 
Maryland 

Doctoral 
students and 
postdoctoral 
fellows 

2002-06 

(no-cost 

extension 

through 

9/2007) 

$1 ,750,000 

Center for Improving Teacher Quality 
(CTQ): Designed to assist states in improving 
the preparation, licensing, and ongoing 
professional development (PD) of teachers of 
students with disabilities through the 
development and implementation of state 
action plans. 

Council of 
Chief State 
School 
Officers 

State agency 
and IHE 
personnel 

2002-07 

$4,983,285 

Center for Early Childhood 
Education/Early Intervention Personnel 
Preparation (EI/ECSE): Created to inform 
EI/ECSE personnel preparation policy and 
practices at all levels through the collection, 
synthesis, and analysis of information on 
certification and licensure, personnel 
preparation, and supply and demand and the 
implementation of research in these areas. 

University of 
Connecticut 

IHEs; local, 
state, and 
national 
governments 

2003-007 

$2,961,124 
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Table 2.2: Overview of PDP National Centers funded in FY 2001 through FY 2007 (continued) 


Center name and purpose 

Grantee 

Intended 

audience 

Funding 

period 

Funding 

amount 

Professional Development in Autism 
Center (PDA): Designed to increase the 
capacity of local school districts to meet the 
needs of students with Autism Spectrum 
Disorder (ASD) through regional and national 
training, technical assistance (TA), and 
education materials. 

University of 
Washington 

State, district, 
and school 
staff 

2003-07 

$5,049,993 

National Center for Leadership in Visual 
Impairments (NCLVI): Designed to increase 
the number of doctoral graduates in special 
education with an emphasis in visual 
impairment (VI) and enhance the training of 
those VI doctoral students through the 
development of a community of practice 
among a consortium of 14 universities. 

Pennsylvania 
College of 
Optometry; 
transferred to 
Salus 

University 

Doctoral 

students 

2004-09 

$5,054,671 

Interdisciplinary Training in Analysis of 
Large-Scale Databases Center (ITALD): 

Designed to increase the number of 
individuals qualified to conduct analyses of 
large-scale datasets related to children and 
youth with disabilities by recruiting, selecting, 
and retaining doctoral students and providing 
an array of coursework and internships. 

University of 
Maryland 

Doctoral 

students 

2006-11 

$2,499,576 

IDEA and Research for Inclusive Settings 
Center for Training Enhancements (IRIS- 
II): Designed to assist college faculty and PD 
providers in the preparation of high-quality 
personnel in inclusive settings through the 
provision of free, online training 
enhancements. 

Vanderbilt 

University 

PD providers 
and IHE faculty 

2006-11 

$6,750,000 

National Institute for Urban School 
Improvement Principal Leadership 
Academies Initiative (NIUSI-PLAN 
LeadScape): Designed to assist school 
principals nationwide in developing and 
implementing inclusive schools to ensure that 
students with and without disabilities meet or 
exceed state academic standards, through the 
use of annual forums, provision of evidence- 
based professional learning modules and on- 
site coaching, and dissemination of materials. 

Arizona State 
University, 
transferred to 
University of 
Arizona 

District and 

school 

personnel, 

especially 

principals 

2006-11 

$1,424,999 
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Table 2.2: Overview of PDP National Centers funded in FY 2001 through FY 2007 (continued) 


Center name and purpose 

Grantee 

Intended 

audience 

Funding 

period 

Funding 

amount 

National Professional Development Center 
on Inclusion (NPDCI): Created to assist 
states in increasing the number of high-quality 
personnel serving preschoolers with 
disabilities in inclusive settings by providing 
ongoing PD, training, and TA. 

University of 
North 
Carolina at 
Chapel Hill, 
Frank Porter 
Graham Child 
Development 
Institute 

State and local 
early childhood 
agency 

administrators; 
faculty and PD 
providers; 
practitioners; 
families 

2006-11 

$2,372,165 

The Center to Mobilize Early Childhood 
Knowledge (CONNECT): Designed to assist 
faculty and PD providers to prepare highly 
qualified personnel in special education, 
related services, early intervention, and early 
care and education to work with young 
children with disabilities and their families, 
through the use of web-based learning 
modules and provision of TA. 

University of 
North 
Carolina at 
Chapel Hill, 
Frank Porter 
Graham Child 
Development 
Institute 

Faculty and PD 
providers 

2007-12 

$2,998,870 

National Center to Inform Policy and 
Practice in Special Education Professional 
Development (NCIPP): Designed to 
recommend policies and practices that 
improve beginning special education teacher 
retention and provide teachers with the 
knowledge and skills to effectively support 
students with disabilities, through research on 
teacher induction and mentoring and 
provision of targeted TA. 

University of 
Florida 

IHEs; state and 
local education 
agencies; 
building 
administrators 

2007-12 

$2,500,000 

National Professional Development Center 
on Autism Spectrum Disorders: Designed as 
a multisite project to promote the use of 
evidence-based practices for children and 
adolescents with ASD by providing training 
and TA to states to increase state capacity to 
support practitioners and deliver sendees for 
children and youth with ASD and their 
families. 

University of 
North 
Carolina at 
Chapel Hill, 
Frank Porter 
Graham 
Child 

Development 

Institute 

State, district, 
and school 
staff; 

practitioners; 

families 

2007-12 

$5,000,000 
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Table 2.2: Overview of PDP National Centers funded in FY 2001 through FY 2007 (continued) 


TABLE READS: The IDEA and Research for Inclusive Settings Center for Faculty Enhancement (IRIS-I), based on a PDP grant 
awarded to Vanderbilt University, was designed to assist college faculty to better prepare general education teachers, 
administrators, school counselors, and school nurses who have key roles in educating students with disabilities through the 
provision of online modules and course enhancement materials. The intended audience included IHE faculty and staff. The center 
was funded from 2001-06 at a total amount of $4,249,995. 

NOTE: The two PDP National Centers listed in this table in italics were funded between FY 2001 and FY 2007, but were not 
included in in-depth data collection for the evaluation because of their similarity to two other National Centers. 

SOURCE: Center grant applications provided by the Office of Special Education Programs (OSEP), U.S. Department of Education, 
and the Online Discretionary Grants Public Database accessed through the OSEP Technical Assistance and Dissemination 
Network. 


Two centers funded during the FY 2001 to FY 2007 time period were not included in the 
in-depth analysis for this study. The National Professional Development Center on Autism 
Spectrum Disorders, which was newly funded at the University of North Carolina in 2007, was 
not included in the study because a sufficient quantity of products was not likely to be available 
in time for inventory and review in 2010. The prior autism- focused center, the Professional 
Development in Autism Center at the University of Washington, was continuing activities with a 
no-cost extension in 2008 and was included in the study. The second center not included in the 
study was Policy Leadership Training in Conducting Large-Scale Special Education Research, 
which was the precursor to the center for Interdisciplinary Training in Analysis of Large-Scale 
Databases. Both were housed at the University of Maryland, with the same project director and 
mission, and funding for the two grants overlapped; the latter grant was judged by the evaluation 
study team to be sufficiently representative of the work of both centers. 

Review of Extant Data 

The evaluation study team reviewed select materials related to the PDP center grant 
program and to the National Centers themselves to inform the design of the evaluation and 
development of data collection instruments. This review included the extraction of information 
from OSEP grant priorities, successful center grant applications, project mid-course reviews 
(“3+2” materials), grant performance reports, and websites and other publicly available center 
products and materials. The extant data were used (1) to prepare for interviews with center 
directors, (2) to guide development of rubric indicators for the expert panel review, and (3) to 
prepare descriptions of each center in combination with information from interviews with center 
directors. In addition, we gathered extant data from the ED Grant Award Database and the online 
Discretionary Grants Public Database, accessed through the OSEP Technical Assistance and 
Dissemination (TA&D) Network website, and analyzed for the number of National Center grants 
awarded from FY 2001 through FY 2007 and the total funding awarded throughout grant funding 
cycles. 

Inventory of National Center Products and Services 

Between August 2008 and March 2010, we contacted the directors of each of the 12 
National Centers selected for the in-depth study and asked them to complete a standardized 
questionnaire (referred to here as an inventory and included as appendix B) to describe each 
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product or service distributed or prepared using PDP grant funds. A product or service was 
defined as a single activity or group of closely related activities designed to achieve a specific 
outcome for a specific audience. We asked center directors to provide a description of each 
product and service developed using grant funds, the presentation medium, approximate cost, 7 
and beginning and end dates for availability. 

The inventory asked center directors to identify up to 10 percent of the center’s products 
or services as signature works. The evaluation study team reasoned that, because the percentage 
of products and services that were to be reviewed by experts was relatively small, it was useful, 
instead of drawing a simple random sample of products and services, to give centers an 
opportunity to identify what they considered their most important PDP-supported work, so that 
those works could be oversampled. The protocol for obtaining an inventory of National Center 
products and services defined signature works as products or services that met the following 
criteria: 

• prominent in terms of the proportion of the National Center’s effort; 

• exclusively or nearly exclusively a product of the National Center’s efforts; 

• predominately supported by PDP grant funds; and 

• met a critical need in the field or to the class of customers served. 

We piloted the inventory of National Center products and services with the IRIS-I center 
in September 2008 and the CILSPRT center in January 2009. We selected these two centers 
because their funding periods (including no-cost extensions) had ended in 2007 and their product 
lists would be complete. Based on feedback from participants, we made minor wording changes, 
and remaining inventories were administered from November 2009 through March 2010. We 
obtained product cost information for IRIS-I during this latter period of time, but we were unable 
to obtain cost information for CILSPRT. In general, with the exception of the pilots, the 
inventory was complete before the center director interview but was not finalized until after that 
interview, since the interview provided an opportunity to resolve questions regarding the 
inventory. 

Inventory data from all 12 centers served as the basis for analysis. We analyzed inventory 
responses to generate a categorized list of products and services, with associated costs, for each 
center as well as across all centers. Initial inventories of products and services were reviewed by 
members of the evaluation study team before the interviews with center directors, and 
clarifications were sought during the interviews. In particular, we worked with center directors to 
group or ungroup items into products or services that could stand alone and were amenable to the 
expert panel review process. Once the lists of products and services were finalized, we developed 
codes for product type. Two members of the evaluation study team independently coded 
products based on data reported on inventories from the first two centers. They then compared 
codes and reconciled any differences to arrive at consensus. One team member then coded the 
products from five centers based on data reported on inventories, while another team member 


7 Center directors were asked to “estimate the cost, in dollars, of developing and delivering the product or service,” including 
overhead. They were instructed to make the estimates “as accurate as reasonably possible” but not to “complete a detailed audit 
of costs per product.” They were also instructed that the sum of costs for all the Center’s products and services “should 
approximate total expenditures to date, excluding costs for the Center’s external evaluator.” For more details, see appendix B. 
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coded the products for the other five centers. Throughout the coding of products for these 
remaining 10 centers, the two team members conducted regular meetings to discuss any items in 
question and come to consensus on the product code when necessary. Counts and distributions of 
products and services, with associated costs, were generated within product types. 

Limitations of the Inventory of Products and Services 

Funding cycles for the centers ranged from 2001-06 to 2007-12. For six of the centers, 
the evaluation focused on all products and services generated during the life of those centers; for 
the other six centers, the evaluation focused on products and services generated by the time of 
data collection, before the end of those grants. Consequently, the inventory of products and 
services did not cover the full period of the grants for half of the National Centers included in the 
evaluation. 

Readers are also cautioned that the costs associated with products and services are based 
on estimates provided retrospectively by the centers rather than a detailed audit of costs per 
product. Additionally, the cost estimates do not reflect costs for all products or services 
developed by the six centers that continued operations after the time of data collection. Finally, 
only 10 of the 12 centers were able to provide costs, and the products and services for which they 
provided costs represented only 51 percent of the products and services the 12 centers identified. 

Interviews with National Center Directors 

Between August 2008 and March 2010, we conducted semi- structured telephone 
interviews with the directors of 12 National Centers. Interview protocols (included as appendix 
A) were piloted with the IRIS-I center in August 2008 and the CILSPRT center in December 
2008. As was the case for the pilot inventories, we selected these two centers because their 
funding had ended in 2007 and their center activities would be complete. Based on feedback 
from participants, minor wording changes were made to the protocols, and the remaining 
interviews were conducted from December 2009 through March 2010. In the interviews with 
center directors, we asked respondents to describe 

• the center’s goals and objectives; 

• the center’s organizational structure and staffing; 

• types of center products, services, or activities used the least/most extensively; 

• the ways in which the work of the center was a continuation of work done previously; 
and 

• obstacles and challenges that the center faced in completing the proposed work. 

Additionally, as noted above, with the exception of the two pilot centers, members of the 
evaluation study team reviewed completed inventories of products and services before the 
interviews. Clarifications were sought during the interviews, and the inventories were revised 
when necessary, following the interviews. 

Interview data from all 12 centers served as the basis for analysis. Members of the 
evaluation study team read through the interview data, organized the data, and highlighted 
information to create an abstract on each center. The abstracts were provided to members of the 
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review panels for the National Centers to orient them to the overall work of the centers and thus 
to provide context for their review of products and services. To avoid biasing the review, the 
abstracts did not provide any specific descriptions of the products or services. 

Expert Review of National Center Products and Services 

An expert panel conducted rubric-based reviews of a sample of center products and 
services produced with PDP grant funding. 

Sampling and Collection of Center Products and Services 

Data from the inventory of National Center products and services were the basis for 
selecting a sample of each center’s products for expert panel review. From each of the 12 
centers, the evaluation study team sampled up to 10 products developed with grant funding, 
including signature works, which were sampled in a separate stratum (up to 3 sampled). To 
develop a sampling frame, we removed from the inventory products that were unsuitable for 
panel review. These included websites, including activities facilitating website use, and 
promotional materials about the availability of TA services, as well as outreach, recruitment, and 
networking activities. We used a random number generator to sample products from the lists of 
signature and nonsignature works for each center, if the frame included more than the required 
number of products in the signature or nonsignature stratum. Centers varied in the types of 
products they generated and the number of products of each type, and not every center listed 10 
products or 3 signature works. As a result, the sample of products reviewed for each center was 
unique in type as well as in content. If a combined signature/nonsignature frame included 10 or 
fewer products, all the products were selected. We collected materials between February and 
April 2010. 

Once the initial sample products were collected, we again screened the suitability of 
products for rubric-based panel review. If a sampled product was deemed unsuitable for review, 
the next product from the randomized list was selected as a replacement. Products were not 
reviewed by expert panels for reasons that included the following: missing or insufficient 
documentation in the judgment of the evaluation study team (for example, a PowerPoint 
presentation only); lack of stability of documentation (as in the case of a website as the only 
documentation); a paper for which a National Center staff member was not the first author listed; 
submission of promotional materials; submission of internal planning documents; or the 
bundling of a product with another product, making a distinct review infeasible. 

Once the final 101 National Center products were selected for review, we asked center 
staff to provide further information on sampled products or services, including evidence of need, 
reasoning behind the design, and evaluative information. Following the collection of the final 
sample products, a member of the evaluation study team who had not previously participated in 
the coding reviewed the product codes for all sampled products based on the actual products 
received. This team member disagreed with the codes of seven of the products, representing 93 
percent agreement. All evaluation study team members who had participated in coding discussed 
the instances of disagreement, reached consensus, and applied final product codes. Expert 
panelists reviewed 101 products (10 products from each of seven centers, 8 products from each 
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of two centers, and 5 products from each of three centers) representing 13 percent of all reported 
products. Fifteen of these products were signature works. 

Review Panels for National Centers 

The evaluation study team recruited 36 experts with relevant content expertise to serve on 
three-person review panels for the 12 National Centers. Panelists were selected to have 
knowledge regarding (1) the content on which the centers focus; (2) appropriate processes for 
effecting changes in, or imparting knowledge to, the adults who are the centers’ target audiences; 
and (3) the relevance and usefulness of the products and services provided by the centers. We 
identified panelists through literature searches, recommendations made by members of the 
evaluation study team and IES staff, and recommendations from potential panelists regarding 
fellow experts. Each potential panelist was identified, recruited, and assigned to review a 
particular center based on expertise relevant to the work of that specific center. Review panel 
members came from academia, state educational agencies, and consulting firms. Members were 
screened to prevent conflicts of interest in the rating of centers. 

In March 2010, the evaluation study team trained all review panel members on the review 
process and the rubrics. The specific purposes of the training were to inform panelists about their 
roles and responsibilities, provide an overview of the review process, and enhance the reliability 
of the ratings. Panelists reviewed the indicators and anchors of the scoring rubrics, practiced 
scoring sample products collected from pilot centers using the rubrics, debriefed with their 
fellow panelists, and practiced reconciliation with their colleagues when ratings differed. 
Panelists also provided feedback on the content of the rubrics. Thirty-four panelists attended a 
daylong, in-person training; two panelists were trained via conference call. 

A total of 101 products were reviewed and rated, up to 10 from each center. One hundred 
products were each reviewed by three experts, while one product was rated by two of the three 
experts assigned to the relevant center because the third expert had a conflict of interest for that 
product. The 302 product reviews took 2.5 hours each, on average, with another hour on average 
for resolution of discrepant ratings on a portion of products. Overall, panelists required an 
average of 26 hours to complete their review process work. Adding 8 hours per reviewer for 
training on the use of the rubrics, the typical reviewer committed 34 hours to the process. 
Reviews were conducted from May through September 2010. 

Rubrics to Rate Center Products and Services 

Panelists used quantitative instruments (included as appendix C) developed by the 
evaluation study team to rate sampled products and services along two dimensions: content 
quality and relevance/usefulness. Because OSEP defined a different content focus for each 
National Center, individual rubrics were developed for each of the 12 centers included in the 
evaluation. For each center, the same rubric was used to rate all of the products reviewed. All of 
the rubrics were li nk ed to research in the relevant content areas and were reviewed by Technical 
Working Group (TWG) members advising the evaluation study team and having corresponding 
content expertise. Eleven of the rubrics were based on the expectations for the center stated in 
the relevant grant priorities developed by OSEP. For these centers, we linked indicators within 
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the rubrics to the expectations and the application review criteria in OSEP’s request for 
applications. One grant, NCLVI, was awarded based on an unsolicited proposal, which means no 
request for applications existed. The corresponding rubric, reviewed by members of the TWG, 
was based on rubrics for the other centers, especially CILSPRT. 

For each center, all products and services submitted for expert panel review were rated 
along the dimensions of quality and relevance/usefulness using the center-specific rubric. The 
specific indicators of quality and relevance/usefulness varied by rubric. Each rubric included 
similar indicators that were expected to be found across centers (e.g., the product or service 
addresses a high-priority need), as well as unique indicators corresponding to the particular focus 
of that center (e.g., the product or service enhances efforts to understand, develop, or implement 
induction and mentoring policies and practices). The number of indicators varied by rubric, with 
the number of quality indicators ranging from 3 to 9 per rubric, and the number of 
relevance/usefulness indicators ranging from 3 to 6 per rubric. Across all 12 rubrics, there were 
76 center-specific indicators of quality, and 55 center-specific indicators of relevance/usefulness. 

Rubric development was informed through external review and testing. In February and 
March 2010, we piloted two draft rubrics. For that pilot, two members of the TWG reviewed the 
rubrics in general and in reference to their application with sample products. These TWG 
members were selected based on their relevant expertise in instrumentation and content areas. 
Each of the pilot reviewers used the relevant draft rubric for that center to rate two sample 
products collected from grantees. They submitted their completed rubrics and provided written 
feedback with regard to the (1) clarity and appropriateness of the indicators and dimensions for 
evaluating the center products, (2) utility of the 5-point scale, (3) difficulty of applying ratings at 
the product/service level, (4) use of “not applicable” (NA) as a rating option, and (5) use of 
reviewer comments on ratings. We used interviews with the pilot reviewers to assess their 
understanding of the rubrics and rationale for their ratings. These interviews informed revisions 
to the rubrics. 

During the March 2010 training of review panel members, panelists practiced scoring 
sample products using the appropriate rubric for each center and provided feedback on rubric 
content and applicability. Panelists discussed each indicator and anchor for their respective 
center and recommended any necessary modifications. This feedback was used to further refine 
the rubrics and prepare them for use in the evaluation. Products used for the development and 
piloting of the rubrics were not those sampled for the actual study. Thus, no scores obtained 
during rubric development or piloting were used in the analysis. 

Final rubrics shared a similar format across centers consisting of (1) quality and 
relevance/usefulness dimensions; (2) indicators rated on a 5-point scale, with 1 being the lowest 
rating and 5 being the highest; (3) availability of NA for indicators for a given product; 

(4) provision of anchor descriptions for scores of 1, 3, and 5; (5) an indicator-specific comment 
field provided to justify NAs and to record raters’ notes; and (6) a general comment field 
provided to indicate uncertainty about a rating or to note strengths or weaknesses not linked to a 
specific indicator. 
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Analysis of Review Panel Data 


Expert panel members rated 101 selected products on a series of indicators grouped into 
two dimensions: quality and relevance/usefulness. Data were derived from expert ratings of 15 
signature products and 86 nonsignature products using 76 quality indicators and 55 
relevance/usefulness indicators. The number of indicators varied by center. Each product was 
rated by three experts. One panelist was exempt from rating one product due to a predetermined 
conflict of interest. If individual experts decided that a particular indicator was not applicable to 
a particular product, they did not rate that indicator (and checked NA). One product was given 
NA for 13 out of 18 indicator-level ratings for relevance/usefulness after reconciliation, and was 
not included in analyses for that dimension. The result was a total of 2,986 ratings: 1,699 ratings 
on indicators of quality and 1,287 ratings on indicators of relevance/usefulness. Broken out by 
signature status, the process generated 463 ratings of signature products (247 quality, 2 1 6 
relevance/usefulness) and 2,523 ratings of nonsignature products (1,452 quality, 1,071 
relevance/usefulness). We calculated quality and relevance/usefulness means as the sum of all 
quality or relevance/usefulness ratings divided by the number of ratings. Quality and 
relevance/usefulness means were calculated by signature status, by product type (i.e., briefs or 
short reports; conferences, institutes, or workshops; intensive, focused TA; etc.), and by center. 
Additionally, we calculated, by signature status and by center, the distributions of quality and 
relevance/usefulness ratings as the percentage of indicator-level ratings at each scale value. 

Inter-rater Agreement for the Review Panels for the National Centers 

The evaluation study team used Rwg(j) (James, Demaree, and Wolf, 1984) as an index 
for estimating the inter-rater agreement of the expert panel review of products and services and 
for identifying cases for reconciliation. Rwg(j) ranges from 0 to 1 and estimates the level of 
inter-rater agreement between judges who provide ratings for multiple items for each target (i.e., 
dimension). Indices were calculated based on uniform distribution assumptions, that is, raters 
who respond randomly to products have an equal chance of choosing each response option on 
the Likert scale. The following standards were used for interpreting Rwg(j) (LeBreton and Senter 
2008): 0.00-0.30 (lack of agreement), 0.31-0.50 (weak agreement), 0.51-0.70 (moderate 
agreement), 0.71-0.90 (strong agreement), 0.91-1.00 (very strong agreement). Additional 
information on the Rwg(j) statistic is provided as appendix F. 

Mean inter-rater agreements based on initial, independent ratings were 0.89 for quality 
(strong agreement) and 0.80 for relevance/usefulness (strong agreement), for an average Rwg(j) 
of 0.84 (strong agreement). Out of the 101 products that were reviewed, 9 had an Rwg(j) of less 
than 0.30 for quality, relevance/usefulness, or both. The evaluation study team convened a 
reconciliation session via conference call for a quality or relevance/usefulness rating of a product 
with an Rwg(j) below 0.30. Seven reconciliation calls were conducted with reference to products 
from six centers. Each call was facilitated by an evaluation study team researcher. Panelists 
discussed the strengths and weaknesses of the product in question and provided rationales for 
their specific ratings on those indicators with the lowest agreement. Experts had the opportunity 
to revisit their ratings and resolve discrepancies, although there was no requirement to reach 
consensus during the reconciliation process. Within 3 business days following the reconciliation 
call, panelists infonned the evaluation study team via email of any revised ratings or that no 
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changes were being made. Following the reconciliation process, three products had an Rwg(j) of 
less than 0.30 on quality, relevance/usefulness, or both. Mean inter-rater agreements were 
increased to 0.91 for quality (very strong agreement) and 0.84 for relevance/usefulness (strong 
agreement), for an average post-reconciliation Rwg(j) of 0.88 (strong agreement). 

As noted above, expert panel reviewers were given the option of choosing NA for a 
specific indicator if they believed that indicator was not applicable to a specific product. When 
NAs were used, panelists were asked to provide an explanation. We examined the prevalence of 
NAs to determine whether the rubrics were successful in capturing the quality and relevance/ 
usefulness of products during independent review. Analysis of the item-level rating data found 
that fewer than 10 percent of indicators were checked as NA. Thus, most reviewers found that 
the items on the provided rubrics were applicable to particular products. In addition, binary 
analyses using Rwg(j) showed that NAs were generally consistent; that is, reviewers tended to 
agree whether particular items were applicable or not to a given product. 

Limitations of Expert Panel Review of National Centers 

Readers are cautioned that the expert panelist ratings of quality and relevance/usefulness 
of PDP National Center products and services only reflect sampled work for which review was 
feasible through the examination of documents or other media. These ratings do not reflect the 
quality or relevance/usefulness of products and services for which documentation was not 
available. In addition, the ratings do not represent National Center products or services after 
March 2010. 
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3. Summary Findings Across PDP National Centers 


The substudy of PDP National Centers provides a descriptive analysis of 12 National 
Centers funded by the PDP between FY 2001 and FY 2007. The study addressed the following 
research questions: 

• What products were developed and services provided by PDP National Centers funded 
between FY 2001 and FY 2007, and at what cost? 

• What were the quality and relevance/usefulness of documented materials and TA 
provided by PDP National Centers funded between FY 2001 and FY 2007? 

Data were gathered through (1) review of relevant extant data, such as grant priorities, 
winning grant applications, center websites, and progress reports; (2) interviews with center 
directors; (3) inventories of center products and services developed with grant funds; and 
(4) expert review of a sample of center products and services. Summary-level findings across 
centers are presented below. Findings for each National Center are provided in Chapter 4. 


Products and Services Generated by PDP National Centers 

As described in Chapter 2, for each of the 12 National Centers on which this evaluation 
focused, the center director submitted a comprehensive inventory of all the products and services 
developed to date with PDP funding. The directors identified 759 products and services (table 
3.1), of which 356 (47 percent) were presentations or webinars, including invited presentations 
by center staff or center-affiliated scholars and center-produced events for infonnation 
dissemination. 

Ten centers were able to report costs for the development and dissemination of 386 
products (table 3.1). Of the $19.5 million in total costs they reported, the largest share (48 
percent) was for the provision of conferences, institutes, or workshops, and the second largest 
share (18 percent) was for web-based training modules and supporting materials. The cost 
figures must be viewed in light of the limitations to these data described in Chapter 2: Cost 
estimates do not reflect costs for all products or services developed by six centers that continued 
operations after the time of data collection, only 10 of the 12 centers were able to provide costs, 
and the products and services for which costs were provided represent only 5 1 percent of the 
products and services the 12 centers identified. Despite the missing data, the relative proportion 
of costs may be representative because 87 percent of the missing data are in two product types 
(presentations or webinars; research papers, articles, dissertations, or book chapters) where costs 
per product/service were relatively low. 
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Table 3.1: PDP National Center products and services, with reported costs and percentage of 
reported costs, by type of product 


Product type 

Number of 
products 1 

Number of 
products 
with cost 
information 

Reported 

costs 2 

Average 
cost per 
product 

Percent of 
reported 
costs 

All products 

759 

386 

$19,509,083 

$50,542 

100 

Briefs or short reports 

71 

47 

301,246 

6,409 

2 

Conferences, institutes, or 
workshops 3 

88 

69 

9,341,040 

135,377 

48 

Intensive, focused technical 

Q 

Q 

825,240 

91,693 

A 

assistance (TA) 4 

y 

y 

L r 

Materials supporting 
universal TA 5 

li 

li 

1,357,937 

123,449 

7 

Presentations or webinars 6 

356 

151 

619,730 

4,104 

3 

Research papers, articles, 
dissertations, or book 
chapters 

147 

27 

230,975 

8,555 

1 

University courses, programs, 
or training models 

5 

4 

1,100,770 

275,193 

6 

Web-based training modules, 
including supporting 
materials 

56 

56 

3,435,091 

61,341 

18 

Websites, including activities 
facilitating website use 

8 

6 

1,852,371 

308,729 

9 

Other products or services 7 

8 

6 

444,683 

74,114 

2 


TABLE READS: Across the PDP National Centers, a total of 759 products had been developed by the time of data collection, 
with a total reported cost of $19,509 million and average cost of $50,542 for 386 of these products. 

1 Six centers were still active at the time of data collection. Counts of products and services in this table include only those 
developed by the time of data collection. 

2 Ten of the 12 centers were able to report costs for some or all of their products/services. Costs associated with monetary support 
for scholars are not included in this table. 

3 Included center-produced events for the purposes of training recipients, providing general TA, or disseminating information to 
targeted recipients. 

4 Involved an individualized, ongoing relationship between the center and the TA recipient. 

5 Included products developed for individuals’ independent use through their own initiative. Does not include products counted in 
other categories that may also serve TA purposes, such as briefs, institutes, presentations, etc. 

6 Included individual, invited presentations by center staff or center-affiliated scholars, as well as individual, center-produced 
events for information dissemination. 

7 Products and services categorized as “other” included promotional and informational materials about center products/services, 
as well as outreach, recruitment, and networking activities. 

SOURCE: Inventory of products and services collected from two center directors in September 2008 and January 2009, and 
remaining center directors November 2009 through March 2010 (appendix A). 
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Panel Ratings of National Center Products and Services 

In the summer of 2010, expert review panels reviewed 101 products/services selected 
across the 12 centers. Up to 10 products/services were reviewed per center, and up to 3 of these 
could be signature works. Signature works, if identified by the center in its inventory, were 
sampled at a higher rate than nonsignature works. Because sufficient materials were not available 
from the centers to make the review of all sampled products feasible, the products reviewed do 
not represent the full inventory of each center’s signature or nonsignature products. Each product 
was rated by three panelists 8 with relevant content expertise using center-specific rubrics 
containing a series of indicators grouped into two dimensions — quality and relevance/usefulness. 
Rating values ranged from 1 (very low) to 5 (very high). The number of products sampled varied 
by center, and the number of indicators for quality and for relevance/usefulness used for the 
review varied based on the rubric for each center. 

For the 15 reviewed products identified by center staff as signature works, 77 percent of 
the quality ratings and 82 percent of the relevance/usefulness ratings for the signature products 
were high or very high (figure 3.1). Across the 86 nonsignature products, 73 percent of quality 
ratings and 68 percent of relevance/usefulness ratings were high or very high. These ratings do 
not generalize to center products or services that lacked sufficient documentation for review, 
such as projects represented only by websites or PowerPoint presentations. 

Among the 1 5 signature products sampled for expert panel review, mean ratings overall 
(table 3.2) exceeded the threshold for “high” (4.0) for both quality (mean = 4.13) and 
relevance/usefulness (mean = 4.25). Four out of seven types of signature products had mean 
ratings of 4.0 or above for both quality and relevance/usefulness: briefs or short reports; 
conferences, institutes, or workshops; university courses, programs, or training models; web- 
based training modules. For the 86 nonsignature products sampled for expert panel review, mean 
ratings overall exceeded the threshold for “high” for quality (mean = 4.1 1, median = 4.0) and 
exceeded the threshold for “moderate” (3.0) for relevance/usefulness (mean = 3.91). Three out of 
eight types of nonsignature products had mean ratings of 4.0 or above for both quality and 
relevance/usefulness: materials supporting universal TA; university courses, programs, or 
training models; web-based training modules. 


One panelist was exempt from rating one product due to a predetermined conflict of interest. 
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Figure 3.1: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature 
products/services of 12 PDP National Centers 


Percent 



Usefulness Usefulness 


Very Low 
Low 

■ Moderate 

■ High 

■ Very High 


Signature products/services 1 
(n=15 products/services, with 
247 quality ratings and 216 
relevance/usefulness ratings) 


Nonsignature products/services 2 
(n=86 products/services for quality, 
n=85 for relevance/usefulness, with 1,452 quality 
ratings and 1,071 relevance/usefulness ratings) 


FIGURE READS: For the 15 signature products/services reviewed by an expert panel, 1 percent of the 247 quality ratings were 
at the level of 1 (very low); 1 percent were 2 (low); 20 percent were 3 (moderate); 39 percent were 4 (high); and 38 percent 
were 5 (very high). 

'Center staff had been asked to identify up to 10 percent of products/services as signature, defined as works that (1) were 
prominent in terns of the proportion of the center’s overall effort, (2) were exclusively or nearly exclusively a product of the 
center’s effort, (3) were predominately supported by PDP funds or unlikely to have occurred without PDP support, and (4) 
made a consequential contribution to the field or customers served. Two centers did not identify any of their products/services 
as signature works and are therefore not included in the distribution of ratings for signature products/services. 

2 One nonsignature product was not included in analyses for relevance/usefulness because it received NAs for 13 out of 18 
indicator-level ratings. One reviewer was exempt from rating one nonsignature product due to a predetermined conflict of 
interest; those data were treated as missing and are not included in the analysis. 

NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The 
number of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular 
product. Detail may not sum to totals due to rounding. 

SOURCE: Expert panel reviews conducted May through September 2010 using rubrics in appendix C. 
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Table 3.2: Mean expert panelist ratings (on a 5-point scale, with 5 being the highest) for 
sampled PDP National Center products/services, by signature/nonsignature 
status, review dimension, and type of product/service 





Quality 


Relevance/usefulness 


#of 

#of 



#of 




products 

ratings 

Mean 

SD 

ratings 

Mean 

SD 

All sampled signature products 

15 

247 

4.13 

0.84 

216 

4.25 

0.85 

Briefs or short reports 

1 

14 

4.50 

0.76 

12 

4.67 

0.65 

Conferences, institutes, or 
workshops 

1 

9 

4.22 

0.44 

15 

4.27 

0.80 

Intensive, focused technical 
assistance (TA) 

1 

21 

3.19 

0.75 

16 

3.75 

0.58 

Materials supporting universal TA 

2 

36 

3.81 

0.86 

25 

3.96 

0.84 

Presentations or webinars 

0 

t 

t 

t 

t 

t 

t 

Research papers, articles, 
dissertations, or book chapters 

3 

64 

4.05 

0.76 

47 

3.91 

0.88 

University courses, programs, or 
training models 

2 

31 

4.23 

0.84 

18 

4.56 

0.70 

Web-based training modules, 
including supporting materials 

5 

72 

4.51 

0.67 

83 

4.51 

0.83 

All sampled nonsignature 
products 

86 

1,452 

4.11 

0.91 

1,071 

3.91 

1.18 

Briefs or short reports 

6 

127 

3.73 

0.87 

89 

2.91 

1.28 

Conferences, institutes, or 
workshops 

7 

121 

4.19 

0.82 

74 

3.97 

1.12 

Intensive, focused TA 

1 

23 

3.35 

0.71 

17 

3.53 

1.01 

Materials supporting universal TA 

4 

45 

4.04 

0.74 

61 

4.33 

0.75 

Presentations or webinars 

7 

127 

3.68 

0.82 

105 

3.55 

0.97 

Research papers, articles, 
dissertations, or book chapters 

42 

696 

4.21 

0.94 

442 

3.95 

1.20 

University courses, programs, or 
training models 

1 

16 

4.50 

0.73 

9 

4.78 

0.67 

Web-based training modules, 
including supporting materials 

18 

297 

4.24 

0.85 

274 

4.20 

1.07 


TABLE READS: The 15 signature products reviewed by the expert panels received 247 ratings for quality and 216 ratings for 
relevance/usefulness; the overall mean rating for quality was 4. 13 ( sd=0.84 ) and the overall mean rating for relevance/ 
usefulness was 4.25 ( sd=0.85 ). 

t No presentations or webinars were categorized by center staff as signature works. 

NOTE: Overall, experts made 463 ratings across the 15 signature products/services and 2,523 ratings across the 86 nonsignature 
products/services. All ratings were on a 5-point scale, with the following values: l=very low, 2=low, 3=moderate, 4=high, 5=very 
high. In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number of 
ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. Two 
centers did not identify any of their products/services as signature works and are therefore not included in composite ratings for 
signature products/services. One product was given NA for 13 out of 18 indicator-level ratings for relevance/usefulness and was 
therefore not included in analyses for that dimension. One reviewer was exempt from rating one product due to a predetennined 
conflict of interest; those data were treated as missing and are not included in the analysis. 

SOURCE: Expert panel reviews, conducted May through September 2010 using rubrics in appendix C. 
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4. Findings for Specific PDP National Centers 


Findings for each of the 12 individual PDP National Centers included in the evaluation are 
presented below. As noted previously, the products reviewed by the expert panels do not 
represent the full inventory of each center’s signature or nonsignature products because sufficient 
materials were not available from the centers to make the review of all sampled products 
feasible. 


IDEA and Research for Inclusive Settings Center for Faculty Enhancement 

IRIS-I was designed to assist college faculty to better prepare general educators, 
education leaders, school nurses, and school counselors with key roles in educating students with 
disabilities (table 4. 1). The center was funded by the PDP from 2001-06 and housed at 
Vanderbilt University. 


Table 4.1: Descriptive information reported by the IDEA and Research for Inclusive 
Settings Center for Faculty Enhancement (IRIS-I) 


Area 

Center-specific description 

Primary center 
strategies 

• Developing and disseminating free course-enhancement materials based on research- 
validated information about working with students with disabilities in inclusive 
settings, including 

a. web-based training modules; 

b. case study units with school-based scenarios that use research-based methods 
or strategies; and 

c. activities with hands-on learning opportunities, class discussions, and 
independent assignments; 

• Working intensively with selected implementation sites to infuse materials into their 
course work and provide an on-site coordinator; and 

• Promoting the use of course enhancement materials available through the project 
website. 

Most frequently used 
products or services 

• Web-based training modules, particularly those focused on behavior, and 

• Case study units with school-based scenarios. 

Link of center 
activities to prior 
work in the field 

• IRIS-1 had no direct precursor in the field, although the Alliance Project, an earlier 
grant on which IRIS-I staff worked, conducted training and outreach with faculty at 
minority institutions to support their ability to write proposals and obtain grants to 
build infrastructure in special education. 

Challenges faced by 
center 

• Securing timely participation of some top researchers was initially challenging, so 
alternates were identified and recruited as needed. 

• Some faculty used the modules as homework assignments without the expected in- 
class discussion, so the module format was adjusted to add a component on “what the 
experts say.” 

• Making modules accessible to individuals with disabilities, particularly visual 
impairments, was a technical challenge. 

SOURCE: Interview with center director, conducted in August 2008 (appendix A); IRIS 3+2 Evaluation Report; and 
inventory of products and services collected from center director in September 2008 (appendix B). 
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Among other products and services, IRIS-I developed a series of web-based training 
modules based on research-validated infonnation about working with students with disabilities in 
inclusive settings. The cost of developing these web-based training modules, including 
supporting materials, was just under $2.0 million (table 4.2). 


Table 4.2: Count of IDEA and Research for Inclusive Settings Center for Faculty 
Enhancement (IRIS-I) products and services and their reported costs 


Product type 

Number of products 

Reported costs 

IRIS-I products/services 

39 

— 

Intensive, focused technical assistance (TA) 

l 1 

$200,000 

Materials supporting universal TA 

3 2 

375,000 

Research papers, articles, dissertations, or book 
chapters 

l 3 

— 

Web-based training modules, including supporting 
materials 

32 4 

1,963,000 

Websites, including activities promoting website use 

l 5 

766,000 

Other products or services 

l 6 

380,000 


TABLE READS: IRIS-I identified 39 products or services developed with PDP funding. 

— Cost data were not provided for this type of product or cannot be calculated because of missing information. 

1 Intensive support to nine IHEs, including training for faculty and a site coordinator supporting use of IRIS materials. 

2 Ten case study units providing practice with school-based scenarios counted as one product, 70 lesson ideas for 
postsecondary instruction counted as one product, and 85 information guides developed by extant sources and linked to IRIS 
website counted as one product. 

3 Findings from pilot study on the use of IRIS modules in college courses. 

4 Web-based interactive training modules based on the Star Legacy Cycle’s 5 stages: The Challenge, Thoughts, Perspectives 
and Resources, Assessment, and Wrap-Up, 32 modules covering 15 content areas. 

5 One website with web resource directory, online dictionary, film search tool, and module media search tool. 

6 General promotional materials to increase awareness of the center’s products and services. 

SOURCE: Inventory of products and services collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed 10 products sampled from 
the inventory of products provided by IRIS-I. Panelists reviewed each selected IRIS-I 
product/service based on four quality indicators and six relevance/usefulness indicators, with 
rating values ranging from 1 (very low) to 5 (very high). Not all indicators were relevant to all 
products. 

For the two reviewed products identified by center staff as signature works, 79 percent of 
the quality ratings and 86 percent of the relevance/usefulness ratings were high or very high 
(figure 4.1). Across the eight nonsignature products, 93 percent of the quality ratings and 84 
percent of the relevance/usefulness ratings were high or very high. The mean ratings for 
signature products were 4.33 for quality and 4.47 for relevance/usefulness, and the mean ratings 
for nonsignature products were 4.57 for quality and 4.36 for relevance/usefulness (table 4.3). 
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Table 4.3 provides additional details regarding the number of indicators, number of ratings, and 
scores for the specific types of products that were reviewed. 


Figure 4.1: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature 
products/services of the IDEA and Research for Inclusive Settings Center for 
Faculty Enhancement (IRIS-I) 


Percent 


100 


80 


60 


40 


20 



21 
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25 


14 


32 








54 


72 


61 









Quality 


Relevance/ 

Usefulness 


Quality 


Relevance/ 

Usefulness 


Very Low 
Low 

■ Moderate 

■ High 

■ Very High 


Signature products/services 1 
(n=2 products/services, with 
24 quality ratings and 
36 relevance/usefulness ratings) 


Nonsignature products/services 2 
(n=8 products/services for quality, with 
96 quality ratings; n=7 for 
relevance/usefulness, with 
125 relevance/usefulness ratings) 


FIGURE READS: For the two signature IRIS-I products reviewed by an expert panel, none of the 24 quality ratings were at the 
level of 1 (very low) or 2 (low); 21 percent were 3 (moderate); 25 percent were 4 (high); and 54 percent were 5 (very high). 

'Center staff members were asked to identify up to 10 percent of the center’s products/services as signature works. Signature 
works were products or services that (1) were prominent in terms of the proportion of the center’s overall effort, (2) were 
exclusively or nearly exclusively a product of the center’s effort, (3) were predominately supported by grant funds or unlikely 
to have occurred without grant support, and (4) made a consequential contribution to the field or to the class of customers 
served. 

2 One nonsignature product was not included in analyses for relevance/usefulness because it received NAs for 13 out of 18 
indicator-level ratings. 

NOTE: Panelists reviewed each selected IRIS-I product/service based on four quality indicators and six relevance/usefulness 
indicators. In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The 
number of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular 
product. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.3: Expert ratings of sampled signature and nonsignature products/services for 
IDEA and Research for Inclusive Settings Center for Faculty Enhancement 
(IRIS-I), by review dimension and type of product/service 


Quality 


IRIS-I products/services 
reviewed 

Number of 
products/services 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Signature 

2 

4 

24 

4.33 

0.82 

6 

36 

4.47 

1.06 

Web-based training modules, 
including supporting 
materials 

2 

4 

24 

4.33 

0.82 

6 

36 

4.47 

1.06 

Nonsignature 

8 

4 

96 

4.57 

0.61 

6 

125 

4.36 

1.07 

Materials supporting 
universal technical assistance 

1 

4 

12 

4.25 

0.75 

6 

17 

4.53 

0.72 

Research papers, articles, 
dissertations, or book 
chapters 

1 

4 

12 

4.83 

0.39 

t 

t 

t 

t 

Web-based training modules, 
including supporting 
materials 

6 

4 

72 

4.58 

0.60 

6 

108 

4.33 

1.12 

TABEE READS: Two IRIS-I signature products 

were reviewed. Each of three experts rated up to four quality indicators for 



Relevance/usefulness 


each product, resulting in 24 ratings. The mean rating for quality was 4.33 ( sd=0.82 ). The experts rated up to six 
relevance/usefulness indicators for each product, resulting in 36 ratings. The mean rating for relevance/usefulness was 4.47 
(sd=1.06). 

| Product was given NA for 13 out of 18 indicator-level ratings for relevance/usefulness, and was therefore not included in for 
this calculation. 


NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
IRIS-I, experts made 286 ratings across the 10 products/services. Five ratings were not included in the analysis for the reason 
stated above (f). All ratings were on a 5-point scale, with the following values: l=very low, 2=low, 3=moderate, 4=high, 5=very 
high. Neither center websites nor products categorized as other were included in the expert review process. 

SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 
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Carolina Interdisciplinary Large-Scale Policy Research Training Center 

CILSPRT was designed to prepare special education doctoral students and postdoctoral 
fellows to conduct large-scale and longitudinal policy studies. The center was funded by the PDP 
from 2002-06 and conducted by the University of North Carolina’s Frank Porter Graham Child 
Development Institute (FPG). The center held onsite training sessions related to large-scale 
research and placed fellows in mentored research apprenticeships (table 4.4). 


Table 4.4: Descriptive information reported by the Carolina Interdisciplinary Large-Scale 
Policy Research Training Center (CILSPRT) 


Area Center-specific description 

Primary center • Recruiting, mentoring, training, and supporting interdisciplinary doctoral and 

strategies postdoctoral fellows in special education to conduct large-scale and 

longitudinal policy studies; 

• Holding annual onsite 8- to 12-week training sessions (called boot camp) to 
provide fellows with the knowledge and competencies needed to conduct 
large-scale research and an understanding of the policy context in which 
these studies are conducted; 

• Supporting fellows in developing proposals to conduct secondary analyses of 
large-scale special education datasets; 

• Placing fellows in a 10-month research apprenticeship with a mentor; and 

» Supporting publication and presentation of fellows’ research. 

Most frequently • Syllabus and packet of readings from boot camp 

used products or 
services 1 

Link of center • At the time of PDP grant application, some staff at Frank Porter Graham 

activities to prior (FPG) were involved in analysis of data from the National Early Intervention 

work in the field Longitudinal Study, and other employees at FPG had done large-scale data 

analysis in the past. However, CILSPRT was not a direct extension of any 

work done previously. 

Challenges faced • Recruiting sufficient numbers of fellows, in part because of the requirement 
by center to relocate to NC (in all, 17 fellows were recruited); 

• Having fellows complete the program in 1 year; and 

• Introducing a course into the school of education, an objective that was 

ultimately accomplished. 

1 This product could be referenced as most central to the center, but not necessarily most frequently used. 

SOURCE: Interview with center director, conducted in December 2008 (appendix A), and Online Discretionary Grants 
Public Database accessed through the Office of Special Education Programs Technical Assistance and Dissemination 
Network. 


CILSPRT fellows conducted a number of presentations and produced a number of 
research papers while enrolled in the program (table 4.5). 
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Table 4.5: Count of Carolina Interdisciplinary Large-Scale Policy Research Training 
Center (CILSPRT) products and services 


Product type 

Number of products 

CILSPRT products/services 

85 

Presentations or webinars 

35 1 

Research papers, articles, dissertations, or book chapters 

48 2 

University courses, programs, or training models 

l 3 

Other products or services 

l 4 


TABLE READS: CILSPRT identified 85 products or services developed with PDP grant funding. 

1 Conference presentations or co-presentations by fellows while enrolled in the program. 

2 Papers authored or co-authored by fellows while enrolled in the program. 

3 An 8- to 12-week training program with presentations and readings. 

4 General promotional materials to increase awareness of the center’s program. 

NOTE: CILSPRT provided no data on costs for products or services. 

SOURCE: Inventory of products and services collected from center director in January 2009 (appendix B). 


In the summer of 2010, three expert panel members reviewed 10 products sampled from 
the inventory of products provided by CILSPRT. Panelists reviewed each selected CILSPRT 
product/service based on seven quality indicators and three relevance/usefulness indicators, with 
rating values ranging from 1 (very low) to 5 (very high). Not all indicators were relevant to all 
products. 

For the one reviewed product identified by center staff as a signature work, 50 percent of 
the quality ratings and 77 percent of the relevance/usefulness ratings were high or very high 
(figure 4.2). Across the nine nonsignature products, 82 percent of the quality ratings and 70 
percent of relevance/usefulness ratings were high or very high. The mean ratings for the 
signature product were 3.55 for quality and 4.22 for relevance/usefulness, and the mean ratings 
for nonsignature products were 4.37 for quality and 4.03 for relevance/usefulness (table 4.6). 
Table 4.6 provides additional details regarding the number of indicators, number of ratings, and 
scores for the specific types of products that were reviewed. 
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Figure 4.2: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature 
products/services of the Carolina Interdisciplinary Large-Scale Policy 
Research Training Center (CILSPRT) 


Percent 

100 

80 
60 
40 


20 


0 


13 


38 



22 



Quality Relevance/ 

Usefulness 

Signature products/services 1 
(n=l product/service, with 11 quality 
ratings and 9 relevance/usefulness ratings) 



Quality 


15 



Very Low 
Low 

i Moderate 
I High 
I Very High 


Relevance/ 

Usefulness 

Nonsignature products/services 
(n=9 products/services; with 142 quality 
ratings and 80 relevance/usefulness ratings) 


FIGURE READS: For the signature CILSPRT product reviewed by an expert panel, none of the 1 1 quality ratings were at the 
level of 1 (very low), 13 percent were 2 (low), 38 percent of ratings were 3 (moderate), 25 percent were 4 (high), and 25 
percent were 5 (very high). 

1 Center staff members were asked to identify up to 10 percent of the center’s products/services as signature works. Signature 
works were products or services that (1) were prominent in terms of the proportion of the center’s overall effort, (2) were 
exclusively or nearly exclusively a product of the center’s effort, (3) were predominately supported by grant funds or unlikely to 
have occurred without grant support, and (4) made a consequential contribution to the field or to the class of customers served. 

NOTE: Panelists reviewed each selected CILSPRT product/service based on seven quality indicators and three 
relevance/usefulness indicators. In general, the number of ratings is equal to three raters x the number of products x the 
number of indicators. The number of ratings was reduced, however, when a rater considered a particular indicator to be “not 
applicable” to a particular product. Detail may not sum to totals due to rounding. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.6: Expert ratings on sampled signature and nonsignature Carolina 

Interdisciplinary Large-Scale Policy Research Training (CILSPRT 
products/services, by review dimension and type of product/service 


Quality 


u 

u 


T3 

o 


CILSPRT 

products/services reviewed 

Number of 

QJ 

f/i 

o 

© 

© 

D. 

Number of 
indicators r; 

Number of 
ratings 

Mean 

SD 

Number of 
indicators 

Number of 
ratings 

Mean 

SD 

Signature 


l 

7 

11 

3.55 

0.93 

3 

9 

4.22 

0.83 

University courses, 
programs, or training models 


l 

7 

11 

3.55 

0.93 

3 

9 

4.22 

0.83 

Nonsignature 

Research papers, articles, 


9 

7 

142 

4.37 

0.87 

3 

80 

4.03 

1.30 

dissertations, or book 
chapters 


9 

7 

142 

4.37 

0.87 

3 

80 

4.03 

1.30 


Relevance/usefulness 


TABLE READS: One CILSPRT signature product was reviewed. Each of three experts rated up to seven quality indicators for 
each product, resulting in 1 1 ratings. The mean rating for quality was 3.55 ( sd=0.93 ). The experts rated up to three 
relevance/usefulness indicators for each product, resulting in nine ratings. The mean rating for relevance/usefulness was 4.22 
(sd=0.83). 

NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
CILSPRT, experts made 242 ratings across the 10 products/services. All ratings were on a 5-point scale, with the following 
values: l=very low, 2=low, 3=moderate, 4=high, 5=very high. 


SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 
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Center for Improving Teacher Quality 

CTQ was designed to assist states in improving the preparation, licensing, and ongoing 
professional development of teachers of students with disabilities. The center was funded by the 
PDP from 2002-07 and was a project of the Council of Chief State School Officers. In addition 
to other products and services, CTQ conducted annual meetings to help state teams plan for 
improvements to teacher preparation and create state-specific action plans to reform their teacher 
quality systems (table 4.7). 

Table 4.7: Descriptive information reported by the Center for Improving Teacher Quality 
(CTQ) 


Area 

Center-specific description 

Primary center 
strategies 

• Assisting states to operationalize the standards developed under the Interstate 
Teacher Assessment and Support Consortium (INTASC) by developing 
action plans or models of teacher preparation, licensure, induction, and 
ongoing professional development (PD) for both general and special 
education teachers of students with disabilities; 

• Hosting annual invitational forums for state leaders; 

• Analyzing state action plans developed during the forums and providing 
feedback to the participating states; 

• Preparing and disseminating materials for supporting states in developing and 
implementing their action plans; 

• Offering a variety of resources to help states move forward with their plans, 
including limited grant funds; brokering of resources, including individual 
experts; and newsletters and dissemination of other CTQ materials; 

• Conducting case studies of a statewide model development, including 
technical assistance and documentation of the change process; and 

• Supporting a community of practice and related website. 

Most frequently 
used products or 
services 

• State action plans; 

• Annual forums and related materials; 

• Community of Practice and electronic mailing list via the CTQ website; 

• Annual meeting of case study states; and 

• Small grants to states. 

Link of center 
activities to prior 
work in the field 

• CTQ was an extension of prior work completed by INTASC, which had 
worked with 34 states since 1987. INTASC provided a forum for states to 
learn and collaborate on the development of compatible educational policies 
on teaching, accountability requirements for teacher preparation, techniques 
to assess teacher performance for licensing and evaluation, and programs to 
enhance teacher PD. 

Challenges faced 
by center 

• With the passage of No Child Left Behind, in the first year of the grant, 
highly qualified teacher standards had to be introduced into state action plans. 

• Turnover occurred in state teams. 

• The context within each state required a unique action plan or process for 
developing and implementing those plans. 


SOURCE: Interview with center director, conducted in December 2009 (appendix A). 
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The provision of conferences, institutes, or workshops accounted for 78 percent of 
product development and dissemination costs during the CTQ grant period (table 4.8). 


Table 4.8: Count of Center for Improving Teacher Quality (CTQ) products and services 
and their reported costs 


Product type 

Number of 
products 

Reported costs 

CTQ products/services 

6 

$5,000,000 

Conferences, institutes, or workshops 

3 1 

3,900,000 

Materials supporting universal technical assistance 

2 2 

800,000 

Websites, including activities promoting website use 

l 3 

300,000 


TABLE READS: CTQ identified six products or services developed with PDP funding and reported costs of $5.0 million 
for product development and dissemination. 

1 Annual meetings of six case study state teams to craft and implement action plans counted as one product; Community of 
Practice with ongoing series of conference calls and shared work on the website counted as one product; and Annual Forum 
meetings for 41 state teams to plan for improvements to teacher preparation counted as one product. 

2 Planning tools and resources for state teams and state action plans. 

3 National Community of Practice on Improving Teacher Quality interactive website with space for resource sharing and 
work spaces for case study states. 

SOURCE: Inventory of products and services collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed five products from the 
inventory of products provided by CTQ. Panelists reviewed each selected CTQ product/service 
based on three quality indicators and five relevance/usefulness indicators, with rating values 
ranging from 1 (very low) to 5 (very high). Not all indicators were relevant to all products. 

For the two reviewed products identified by center staff as signature works, 72 percent of 
the quality ratings and 80 percent of the relevance/usefulness ratings were high or very high 
(figure 4.3). Across the three nonsignature products, 78 percent of quality ratings and 57 percent 
of relevance/usefulness ratings for nonsignature products were high or very high. The mean 
ratings for signature products were 3.83 for quality and 4.13 for relevance/usefulness, and the 
mean ratings for nonsignature products were 3.83 for quality and 3.63 for relevance/usefulness 
(table 4.9). Table 4.9 provides additional details regarding the number of indicators, number of 
ratings, and scores for the specific types of products that were reviewed. 
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Figure 4.3: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature 
products/services of the Center for Improving Teacher Quality (CTQ) 
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FIGURE READS: For the two signature CTQ products reviewed by an expert panel, none of the 18 quality ratings were at the 
level of 1 (very low) or 2 (low), 28 percent were 3 (moderate), 61 percent were 4 (high), and 1 1 percent were 5 (very high). 

1 Center staff members were asked to identify up to 10 percent of the center’s products/services as signature works. Signature 
works were products or services that (1) were prominent in terms of the proportion of the center’s overall effort, (2) were 
exclusively or nearly exclusively a product of the center’s effort, (3) were predominately supported by grant funds or unlikely 
to have occurred without grant support, and (4) made a consequential contribution to the field or to the class of customers 
served. 

NOTE: Panelists reviewed each selected CTQ product/service based on three quality indicators and five relevance/usefulness 
indicators. In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The 
number of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular 
product. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.9: Expert ratings on sampled signature and nonsignature Center for Improving 
Teacher Quality (CTQ) products/services, by review dimension and type of 
product/service 


Quality 


CTQ products/services 
reviewed 
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Mean 


SD 


Signature 

2 

3 

18 

3.83 

0.62 

5 

30 

4.13 

0.73 

Conferences, institutes, or 
workshops 

1 

3 

9 

4.22 

0.44 

5 

15 

4.27 

0.80 

Materials supporting 
universal technical assistance 
(TA) 

1 

3 

9 

3.44 

0.53 

5 

15 

4.00 

0.65 

Nonsignature 

3 

3 

23 

3.83 

0.49 

5 

44 

3.68 

1.12 

Conferences, institutes, or 
workshops 

2 

3 

14 

3.79 

0.58 

5 

29 

3.41 

1.18 

Materials supporting 
universal TA 

1 

3 

9 

3.89 

0.33 

5 

15 

4.20 

0.77 


TABLE READS: Two CTQ signature products were reviewed. Each of three experts rated up to three quality indicators for 
each product, resulting in 18 ratings. The mean rating for quality was 3.83 ( sd=0.62 ). The experts rated up to five relevance/ 
usefulness indicators for each product, resulting in 30 ratings. The mean rating for relevance/usefulness was 4.13 (sd=0.73). 

NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
CTQ, experts made 115 ratings across the 5 products/services. All ratings were on a 5-point scale, with the following values: 
l=very low, 2=low, 3=moderate, 4=high, 5=very high. Center websites were not included in the expert review process. 

SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 


34 



Center for Early Childhood Education/Early Intervention Personnel Preparation 

EI/ECSE was created to identify critical gaps in knowledge regarding personnel 
preparation in early intervention and early childhood special education and conduct a program of 
research at the national, state, and local levels to address the identified gaps. The center was 
funded by the PDP from 2003-07 and housed at the University of Conneticut (table 4. 10). 


Table 4.10: Descriptive information reported by the Center for Early Childhood 
Education/Early Intervention Personnel Preparation (EI/ECSE) 


Area 

Primary center 
strategies 


Center-specific description 

• Designing and conducting a program of research at the national, state, 
institutional, and direct provider levels to address gaps in knowledge 
regarding personnel preparation in early intervention and early childhood; 

• Compiling a comprehensive database of current licensure and certification 
standards for all EI/ECSE personnel; 

• Developing a comprehensive profile of current training programs for all types 
of personnel at the institutional, state, and national levels; 

• Describing the current and projected supply and demand for personnel; and 

• Developing and disseminating recommendations regarding personnel 
preparation policy and practice based on research findings. 


Challenges faced • Data collection was more extensive than anticipated due to difficulty 
by center accessing data in the institutions that conduct personnel preparation and 

training. 


NOTE: There was not suffcient information available to report the most frequently used products or services of the center 
or the link of center activities to prior work in the field. 

SOURCE: Interview with center director, conducted in January 2010 (appendix A). 


Among other activities, the center conducted 10 research studies during the period of 
PDP grant funding, resulting in a series of practice and policy guides, data reports, presentations, 
and journal articles (table 4.1 1). 
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Table 4.11: Count of Center for Early Childhood Education/Early Intervention Personnel 
Preparation (EI/ECSE) products and services 


Product type 

Number of products 

EI/ECSE products/services 

58 

Briefs or short reports 

24 1 

Presentations or webinars 

10 2 

Research papers, articles, dissertations, or book chapters 

22 3 

Websites, including activities promoting website use 

l 4 

Other products or services 

l 5 


TABLE READS: EI/ESCE identified 58 products or services developed with PDP funding. 

1 Two Policy Recommendations Think Tanks and 22 At a Glance Briefs. 

2 Nine presentations and one podcast. 

3 Two research articles and 20 papers. 

4 One website with access to presentations, journal articles, data reports, and practice/policy guides. 

5 General promotional materials to promote awareness of center’s products and services. 

NOTE: EI/ECSE provided no data on costs for products or services. 

SOURCE: Inventory of products and services collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed eight products sampled 
from the inventory of products provided by EI/ECSE. Panelists reviewed each EI/ECSE 
product/service based on six quality indicators and four relevance/usefulness indicators, with 
rating values ranging from 1 (very low) to 5 (very high). Not all indicators were relevant to all 
products. 

For the one reviewed product identified by center staff as a signature work, 53 percent of 
the quality ratings and 75 percent of the relevance/usefulness ratings were high or very high 
(figure 4.4). Across the seven nonsignature products, 62 percent of quality ratings and 65 percent 
of relevance/usefulness ratings were high or very high. The mean ratings for the signature 
product were 3.71 for quality and 4.00 for relevance/usefulness, and the mean ratings for 
nonsignature products were 3.76 for quality and 3.89 for relevance/usefulness (table 4.12). Table 
4. 12 provides additional details regarding the number of indicators, number of ratings, and scores 
for the specific types of products that were reviewed. 
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Figure 4.4: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature products/ 
services of the Center for Early Childhood Education/Early Intervention 
Personnel Preparation (EI/ECSE) 
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FIGURE READS: For the signature EI/ECSE product reviewed by an expert panel, none of the 17 quality ratings were at the 
level of 1 (very low) or 2 (low), 47 percent were 3 (moderate), 35 percent were 4 (high) and 18 percent were 5 (very high). 

1 Center staff members were asked to identify up to 10 percent of the center’s products/services as signature works. Signature 
works were products or services that (1) were prominent in terms of the proportion of the center’s overall effort, (2) were 
exclusively or nearly exclusively a product of the center’s effort, (3) were predominately supported by grant funds or unlikely to 
have occurred without grant support, and (4) made a consequential contribution to the field or to the class of customers served. 
Neither center websites nor products categorized as other were included in expert panel reviews. 

NOTE: Panelists reviewed each selected EI/ECSE product/service based on six quality indicators and four relevance/usefulness 
indicators. In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The 
number of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular 
product. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.12: Expert ratings on sampled signature and nonsignature Center for Early 

Childhood Education/Early Intervention Personnel Preparation (EI/ECSE) 
products/services, by review dimension 






Quality 


Relevance/usefulness 

EI/ECSE products/services 
reviewed 

Number of 

products/services 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Number of 
indicators 

Number of 
ratings 

Mean 

SD 

Signature 


1 

6 

17 

3.71 

0.77 

4 

12 

4.00 

0.74 

Research papers, articles, 











dissertations, or book 
chapters 


1 

6 

17 

3.71 

0.77 

4 

12 

4.00 

0.74 

Nonsignature 


7 

6 

121 

3.76 

0.71 

4 

83 

3.89 

1.00 

Research papers, articles, 











dissertations, or book 
chapters 


7 

6 

121 

3.76 

0.71 

4 

83 

3.89 

1.00 


TABLE READS: One EI/ECSE signature product was reviewed. Each of three experts rated up to six quality indicators for 
each product, resulting in 17 ratings. The mean rating for quality was 3.71 ( sd=0 . 77). The experts rated up to four 
relevance/usefulness indicators for each product, resulting in 12 ratings. The mean rating for relevance/usefulness was 4.00 
(sd=0.74). 

NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
EI/ECSE, experts made 233 ratings across the eight products/services. All ratings were on a 5-point scale, with the following 
values: l=very low, 2=low, 3=moderate, 4=high, 5=very high. Neither center websites nor products categorized as other were 
included in the expert review process. 

SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 
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Professional Development in Autism Center 

PDA was designed to increase the capacity of local school districts to meet the needs of 
students with autism spectrum disorders. The center was funded by the PDP from 2003-07 and 
involved six partner institutions: University of Washington, University of Kansas, Maryland 
Coalition for Inclusive Education, University of Colorado-Denver, University of South Florida, 
and Ohio State University. The center provided training on evidence-based practices for state and 
local teams as part of summer institutes and through an online course (table 4.13). 


Table 4.13: Descriptive information reported by the Professional Development in Autism 
Center (PDA) 


Area 

Center-specific description 

Primary center 
strategies 

• Holding intensive trainings on evidence-based practices for state and local 
teams and as part of summer institutes; 

• Providing post-training follow-up and support for up to 1 year; 

• Producing an interactive website with an online course, tip sheets and short 
procedural reviews, research briefs, and consumer guides; and 

• Offering a hank of consultants for assistance with requests. 

Most frequently 
used products or 
services 

• Trainings; 

• Technical assistance tools; and 

• Research briefs. 

Link of center 
activities to prior 
work in the field 

• Support for professional develoment in autism had previously not been done 
on a similar scale. 

Challenges faced 
by center 

• There were obstacles in coordinating the work of 6 partner institutions and 20 
staff members spread across those institutions. 

SOURCE: Interview with center director, conducted in December 2009 (appendix A), and inventory of products and 
services collected from center director between November 2009 and March 2010 (appendix B). 

In addition to other products and services, PDA conducted conferences, institutes, or 
workshops in 23 states, which accounted for 94 percent of costs for product development and 
dissemination during the grant period (table 4.14). 
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Table 4.14: Count of Professional Development in Autism Center (PDA) products and 
services and their reported costs 


Product type 

Number of 
products 

Reported costs 

PDA products/services 

40 

— 

Briefs or short reports 

8 1 

$100,000 

Conferences, institutes, or workshops 

23 2 

4,250,000 

Materials supporting universal technical assistance 

l 3 

100,000 

Research papers, articles, dissertations, or book chapters 

6 4 

6,000 

Web-based training modules, including supporting 

l 5 

50 000 

materials 


D ^ V/Uv 

Websites, including activities promoting website use 

l 6 

— 


TABLE READS: PDA identified 40 products or services developed with PDP funding. 

— Cost data were not provided for this type of product or cannot be calculated because of missing information. 

1 Four research briefs and four practitioner tip sheets. 

2 Districts in at least 23 states participated in one or more training. More precise counts of trainings are unavailable. 

3 Quality indicator assessment-classroom evaluation tool. 

4 Four journal articles and two book chapters. 

5 Autism 101 online course. 

6 One website with access to briefs, presentations, and online course. 

SOURCE: Inventory of products and services collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed eight products sampled 
from the inventory of products provided by PDA. Center staff did not identify any signature 
works, so the study team categorized all eight products as nonsignature works. Panelists 
reviewed each PDA product/service based on seven quality indicators and six relevance/ 
usefulness indicators, with rating values ranging from 1 (very low) to 5 (very high). Not all 
indicators were relevant to all products. 

For the eight nonsignature products, 67 percent of quality ratings and 39 percent of 
relevance/usefulness ratings were high or very high (figure 4.5). The mean ratings were 3.84 for 
quality and 2.86 for relevance/usefulness (table 4.15). Table 4.15 provides additional details 
regarding the number of indicators, number of ratings, and scores for the specific types of 
products that were reviewed. 
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Figure 4.5: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for nonsignature products/services of the 
Professional Development in Autism Center (PDA) 
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FIGURE READS: For the eight PDA products/services reviewed by an expert panel, 1 percent of the 160 quality ratings 
were at the level of 1 (very low), 5 percent were 2 (low), 27 percent were 3 (moderate), 42 percent were 4 (high), and 25 
percent were 5 (very high). 

NOTE: Center staff did not identify any of the center’s products/services as signature works, defined as products or services 
that (1 ) were prominent in terms of the proportion of the center’s overall effort, (2) were exclusively or nearly exclusively a 
product of the center’s effort, (3) were predominately supported by grant funds or unlikely to have occurred without grant 
support, and (4) made a consequential contribution to the field or to the class of customers served. Consequently, the study 
team categorized all 8 products/services as nonsignature works. Center websites were not included in the expert review 
process. Some products and services related to summer institutes, leadership training, and state team trainings were 
unavailable for expert panel review because the center director no longer had access to those materials. Panelists reviewed 
each selected PDA product/service based on seven quality indicators and six relevance/usefulness indicators. In general, the 
number of ratings is equal to three raters x the number of products x the number of indicators. The number of ratings was 
reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.15: Expert ratings on sampled Professional Development in Autism Center (PDA) 
products/services, by review dimension and type of product/service 





Quality 


Relevance/usefulness 

PDA products/services 
reviewed 

Number of 
products/services 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Number of 
indicators 

Number of 
ratings 

Mean 

SD 

Signature 

0 

t 

t 

t 

t 

t 

t 

t 

t 

Nonsignature 

8 

7 

160 

3.84 

0.90 

6 

135 

2.86 

1.33 

Briefs or short reports 

4 

7 

84 

3.82 

0.88 

6 

68 

2.75 

1.33 

Research papers, articles, 
dissertations, or book 
chapters 

3 

7 

56 

4.04 

0.93 

6 

50 

3.02 

1.38 

Web-based training modules, 
including supporting 
materials 

1 

7 

20 

3.40 

0.75 

6 

17 

2.82 

1.24 


TABLE READS: None of the PDA products had been categorized by center staff as a signature work. Eight nonsignature PDA 
products were reviewed. Each of three experts rated up to seven quality indicators for each product, resulting in 160 ratings. The 
mean rating for quality was 3.84 ( sd=0.90 ). The experts rated up to six relevance/usefulness indicators for each product, 
resulting in 135 ratings. The mean rating for relevance/usefulness was 2.86 (sd=1.33). 

tCenter staff did not identify any products/services signature works, so all 8 products/services were categorized as nonsignature 
works. 


NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
PDA, experts made 295 ratings across the 8 products/services. All ratings were on a 5-point scale, with the following values: 
l=very low, 2=low, 3=moderate, 4=high, 5=very high. Center websites were not included in the expert review process. Some 
products and services related to summer institutes, leadership training, and state team trainings were unavailable for expert panel 
review because the center director no longer had access to those materials. 

SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 
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The National Center for Leadership in Visual Impairment 

NCLVI was designed to increase the number of doctoral graduates in special education 
with an emphasis in visual impairment (VI). The center, funded by the PDP from 2004-09, was 
initially housed at the Pennsylvania College of Optometry, then transferred to Salus University. 
The center involved a consortium of 14 universities participating in a collaborative model for 
enhancing the training of leadership personnel in VI and blindness (table 4.16). 


Table 4.16: Descriptive information reported by the National Center for Leadership in 
Visual Impairments (NCLVI) 


Area 

Center-specific description 

Primary center 
strategies 

• Developing a collaborative model for producing leadership personnel in 
visual impairments (VI); 

• Recruiting and providing financial support to students at Consortium 
Member Universities, including tuition, stipends, and expenses for attending 
selected conferences and meetings; 

• Creating enrichment activities and opportunities for in-person research and 
short- and long-term field experiences and internships; and 

• Developing issue-based special topic seminars, on-line discussions, 
specialized lectures, cohort electronic mailing lists, and a community of 
practice for leadership in blindness and VI. 

Challenges faced 
by center 

• Managing 5 years of funding for a 6-year project with two cohorts of students 
required finding ways to save money in order to pay full tuition and stipends 
for the second cohort of doctoral students. 


NOTE: There was not suffcient information available to report the most frequently used products or services of the center 
or the link of center activities to prior work in the field. 

SOURCE: Interview with center director conducted in January 2010 (appendix A), and inventory of products and services 
collected from center director between November 2009 and March 2010 (appendix B). 


NCLVI developed a Collaborative Leadership Training Model and Enrichment Program 
to enhance the preparation of 2 1 doctoral students who produced a number of workshops, 
presentations, and research studies while in the program (table 4.17). 
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Table 4.17: Count of National Center for Leadership in Visual Impairments (NCLVI) 
products and services and their reported costs 


Product type 

Number of 
products 

Reported costs 

NCLVI products/services 

190 

— 

Conferences, institutes, or workshops 

19 1 

— 

Presentations or webinars 

1 5 1 2 

— 

Research papers, articles, dissertations, or book chapters 

18 3 

— 

University courses, programs, or training models 

2 4 

$1,094,000 


TABLE READS: NCLVI identified 190 products or services developed with PDP funding. 

— Cost data were not provided for this type of product or cannot be calculated because of missing information. 

'Two guest lectures, 6 seminars, and 1 1 workshops provided by doctoral fellows. 

2 One hundred twenty-nine conference presentations and 22 poster sessions by doctoral fellows. 

3 Nine dissertations completed and nine dissertations in process at the time of data collection. 

4 NCLVI Enrichment Program: Costs subsumed in this product include courses, administrative costs, faculty salaries, 
seminar agendas, discussion boards, meeting minutes, and external evaluation associated with the program over 4 years; 
Collaborative Leadership Training Model : Costs subsumed in this product include 14-member university consortium, 
including mission statement; guiding principles, public advisory board, committees; worksheets; meeting minutes, and 
evaluations. 

NOTE: NCLVI reported approximately $2,266 million in costs associated with scholarships, stipends, and other monetary 
support for 21 doctoral students. Those costs are not included in this table. Cost data have been rounded to the nearest 
thousand. 

SOURCE: Inventory of products and services collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed 10 products from the 
inventory of products provided by NCLVI. Panelists reviewed each selected NCLVI 
product/service based on seven quality indicators and three relevance/usefulness indicators, with 
rating values ranging from 1 (very low) to 5 (very high). Not all indicators were relevant to all 
products. 

For the one reviewed product identified by center staff as a signature work, all of the 
quality ratings and all of the relevance/usefulness ratings were high or very high (figure 4.6). 
Across the nine nonsignature products, 61 percent of quality ratings and 65 percent of relevance/ 
usefulness ratings were high or very high. The mean ratings for the signature product were 4.60 
for quality and 4.89 for relevance/usefulness, and the mean ratings for nonsignature products 
were 3.80 for quality and 3.91 for relevance/usefulness (table 4.18). Table 4.18 provides 
additional details regarding the number of indicators, number of ratings, and scores for the 
specific types of products that were reviewed. 
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Figure 4.6: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature products/ 
services of the National Center for Leadership in Visual Impairments (NCLVI) 


Percent 



Usefulness Usefulness 


Very Low 
Low 

■ Moderate 

■ High 

■ Very High 


Signature products/services 1 Nonsignature products/services 2 

(n=l product/service, with 20 quality (n=9 products/services, for 115 quality ratings 

ratings and 9 relevance/usefulness ratings) and 78 relevance/usefulness ratings) 


FIGURE READS: For the signature NCLVI product reviewed by an expert panel, none of the 20 quality ratings were at the 
level of 1 (very low), 2 (low), or 3 (moderate); 40 percent were 4 (high); and 60 percent were 5 (very high). 

1 Center staff members were asked to identify up to 10 percent of the center’s products/services as signature works, defined as 
products or services that (1) were prominent in terns of the proportion of the center’s overall effort, (2) were exclusively or 
nearly exclusively a product of the center’s effort, (3) were predominately supported by grant funds or unlikely to have 
occurred without grant support, and (4) made a consequential contribution to the field or to the class of customers served. 

2 One reviewer was exempt from rating one nonsignature product due to a predetermined conflict of interest; those data were 
treated as missing and are not included in the analysis. 

NOTE: Panelists reviewed each selected NCLVI product/service based on seven quality indicators and three 
relevance/usefulness indicators. In general, the number of ratings is equal to three raters x the number of products x the 
number of indicators. The number of ratings was reduced, however, when a rater considered a particular indicator to be “not 
applicable” to a particular product. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.18: Expert ratings on sampled signature and nonsignature National Center for 
Leadership in Visual Impairments (NCLVI) products/services, by review 
dimension and type of product/service 





Quality 


Relevance/usefulness 

NCLVI products/services 
reviewed 

Number of 
products/services 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Number of 
indicators 

Number of 
ratings 

Mean 

SD 

Signature 

1 

7 

20 

4.60 

0.50 

3 

9 

4.89 

0.33 

University courses, 
programs, or training models 

1 

7 

20 

4.60 

0.50 

3 

9 

4.89 

0.33 

Nonsignature 

Research papers, articles, 

9 

5 

115 

3.80 

1.19 

3 

78 

3.91 

1.25 

dissertations, or book 
chapters 

9 

5 

115 

3.80 

1.19 

3 

78 

3.91 

1.25 


TABLE READS: One NCLVI signature product was reviewed. Each of three experts rated up to seven quality indicators for 
each product, resulting in 20 ratings. The mean rating for quality was 4.60 ( sd=0.50 ). The experts rated up to three 
relevance/usefulness indicators for each product, resulting in nine ratings. The mean rating for relevance/usefulness was 4.89 
(sd=0.33). 


NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
NCLVI, experts made 222 ratings across the 10 products/services. All ratings were on a 5-point scale, with the following values: 
l=very low, 2=low, 3=moderate, 4=high, 5=very high. One reviewer was exempt from rating one nonsignature product due to a 
predetennined conflict of interest; those data were treated as missing and are not included in the analysis. 


SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 
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Interdisciplinary Training in Analysis of Large-Scale Databases Center 

ITALD was designed to increase the number of individuals qualified to conduct analyses 
of large-scale datasets related to children and youth with disabilities. The center was funded from 
2006-1 1 and housed at the University of Maryland. The center developed and administered a 
training program to enhance the knowledge and skills of doctoral students conducting large-scale 
special education research (table 4.19). 


Table 4.19: Descriptive information reported by the Interdisciplinary Training in Analysis 
of Large-Scale Databases Center (ITALD) 


Area 

Center-specific description 

Primary center 
strategies 

• Developing and administering a training program to enhance doctoral 
students’ knowledge and skills in large-scale special education research, 
including a doctoral course on policy issues affecting individuals with 
disabilities, monthly group meetings, and training seminars; 

• Providing doctoral student stipends and support for attending conferences 
and fee-based short courses; 

• Conducting workshops for doctoral students and faculty on specific studies 
with data sets available for secondary analysis; and 

• Conducting institutes for local education agencies on how to analyze data to 
answer specific policy questions. 

Link of center 
activities to prior 
work in the field 

• The grant was a continuation of activities occurring during an earlier PDP 
grant, with the exception that the prior grant included postdoctoral fellows. 

Challenges faced 
by center 

• None reported. 


NOTE: ITALD was funded through 2011; information in the table reflects center activities at the time of data collection. 
There was not suffcient infonnation available to report the most frequently used products or services of the center. 
SOURCE: Interview with center director, conducted in December 2009 (appendix A). 


ITALD provided coursework, intensive workshops, and training seminars to doctoral 
students related to the use of large-scale databases, which accounted for 94 percent of reported 
product development and dissemination costs at the time of data collection (table 4.20). 
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Table 4.20: Count of Interdisciplinary Training in Analysis of Large-Scale Databases 
Center (ITALD) products and services and their reported costs 


Product type 

Number of 
products 

Reported costs 

ITALD products/services 

50 

— 

Conferences, institutes, or workshops 

6 1 

$371,000 

Presentations or webinars 

9 2 

— 

Research papers, articles, dissertations, or book chapters 

31 3 

— 

University courses, programs, or training models 

2 4 

6,700 

Other products or services 

2 5 

24,000 


TABLE READS: ITALD identified 50 products or services developed with PDP grant funding at the time of data collection. 
— Cost data were not provided for this type of product or cannot be calculated because of missing information. 

1 Five 2-day training seminars on using various large-scale data sets, including the Special Education Elementary 
Longitudinal Study, the National Longitudinal Transition Study-2 ,and the National Early Intervention Longitudinal Study; 
one-day institute for local education agencies on analyzing Individuals with Disabilities Education Act Section 618 data and 
data required under No Child Left Behind. 

2 Presentations conducted by grant-funded doctoral students. 

3 Papers, journal articles, and dissertations written by grant-funded doctoral students. 

4 Special education policy courses and training model for development of grant-funded doctoral students. 

5 Monthly group meetings with graduate students; recruitment materials. 

NOTE: ITALD was funded through 2011; products and services in this table include only those developed by the time of 
data collection. Costs associated with scholarships, stipends, and other monetary supports for doctoral students are not 
included in this table. Cost data have been rounded to the nearest thousand. 

SOURCE: Inventory of products and services collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed 10 products sampled from 
the inventory of products provided by ITALD. The center staff did not identify any products as 
signature works, so the study team categorized all 10 products as nonsignature works. Panelists 
reviewed each selected ITALD product/service based on seven quality indicators and three 
relevance/usefulness indicators, with rating values ranging from 1 (very low) to 5 (very high). 
Not all indicators were relevant to all products. 

Across the 10 reviewed nonsignature products, 95 percent of quality ratings and 91 
percent of relevance/usefulness ratings were high or very high (figure 4.7). The mean ratings for 
nonsignature products were 4.79 for quality and 4.72 for relevance/usefulness (table 4.21). Table 
4.21 provides additional details regarding the number of indicators, number of ratings, and scores 
for the specific types of products that were reviewed. 
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Figure 4.7: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for nonsignature products/services of the 
Interdisciplinary Training in Analysis of Large-Scale Databases Center 
(ITALD) 


Percent 
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Quality 


Relevance/ Usefulness 


Nonsignature products/services 
(n=10 products/services, with 197 quality 
ratings and 87 relevance/usefulness ratings) 


Very Low 
Low 

■ Moderate 

■ High 

■ Very High 


FIGURE READS: For the ITALD products reviewed by an expert panel, none of the 197 quality ratings were at the level of 1 
(very low) or 2 (low), 5 percent were 3 (moderate), 10 percent were 4 (high), and 85 percent were 5 (very high). 

NOTE: This figure includes data for all reviewed ITALD products. Center staff did not identify any of the center’s 
products/services as signature works, defined as products or services that (1) were prominent in terns of the proportion of the 
center’s overall effort, (2) were exclusively or nearly exclusively a product of the center’s effort, (3) were predominately 
supported by grant hinds or unlikely to have occurred without grant support, and (4) made a consequential contribution to the 
field or to the class of customers served. Consequently, the study team categorized all 10 products/services as nonsignature 
works. Panelists reviewed each selected ITALD product/service based on seven quality indicators and three relevance/usefulness 
indicators. In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The 
number of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular 
product. Detail may not sum to totals due to rounding. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.21: Expert ratings on sampled Interdisciplinary Training in Analysis of Large- 
Scale Databases (ITALD) products/services, by review dimension and type of 
product/service 





Quality 


Relevance/usefulness 

ITALD products/services 
reviewed 

Number of 
products/services 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Number of 
indicators 

Number of 
ratings 

Mean 

SD 

Signature 

0 

t 

t 

t 

t 

t 

t 

t 

t 

Nonsignature 

10 

7 

197 

4.79 

0.52 

3 

87 

4.72 

0.60 

Conferences, institutes, or 

3 

7 

55 

4.62 

0.68 

3 

25 

4.80 

0.50 

workshops 

Research papers, articles, 










dissertations, or book 
chapters 

6 

7 

126 

4.90 

0.34 

3 

53 

4.68 

0.64 

University courses, 
programs, or training models 

1 

7 

16 

4.50 

0.73 

3 

9 

4.78 

0.67 


TABLE READS: None of the ITALD products had been categorized by center staff as a signature work. Ten nonsignature 
ITALD products were reviewed. Each of three experts rated up to seven quality indicators for each product, resulting in 197 
ratings. The mean rating for quality was 4.79 ( sd=0.52 ). The experts rated up to three relevance/usefulness indicators for each 
product, resulting in 87 ratings. The mean rating for relevance/usefulness was 4.72 (sd=0.60). 

fCenter staff did not identify any products/services signature works, so all 10 products/services were categorized as 
nonsignature works. 

NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
ITALD, experts made 284 ratings across the 10 products/services. All ratings were on a 5-point scale, with the following values: 
l=very low, 2=low, 3=moderate, 4=high, 5=very high. Center websites were not included in the expert review process. Some 
products and services related to summer institutes, leadership training, and state team trainings were unavailable for expert panel 
review because the center director no longer had access to those materials. Products categorized as other were not included in the 
expert review process. 

SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 
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IDEA and Research for Inclusive Settings Center for Training Enhancements 


IRIS-II was designed to assist college faculty and professional development providers in 
the preparation of high-quality personnel in inclusive settings. The center was funded from 2006- 
1 1 and housed at Vanderbilt University. The center developed and disseminated free, online 
modules and lesson ideas for use in preservice and in-service training (table 4.22). 


Table 4.22: Descriptive information reported by the IDEA and Research for Inclusive 
Settings Center for Training Enhancements (IRIS-II) 


Area 

Center-specific description 

Primary center 
strategies 

• Developing and disseminating free, online training enhancement materials 
for education and related service faculty and professional development 
providers to use in preservice and in-service training, including 

a. interactive modules; 

b. case study units with school-based scenarios that use research-based 
methods or strategies; and 

c. activities with hands-on learning opportunities, class discussions, and 
independent assignments. 

• Providing training and technical assistance (TA) to facilitate use of center 
materials, including conference presentations, workshops, seminars, and 
summer institutes. 

Most frequently 
used products or 
services 

• Online modules, particularly those focused on cultural and linguistic 
differences and on behavior; and 

• Case study units. 

Link of center 
activities to prior 
work in the field 

• IRIS-II was an expansion of work by the same grantees done under a prior 
PDP National Center, IRIS-I. IRIS-I focused only on non-special education 
faculty and did not include provisions for training and TA. 

Challenges faced 
by center 

• At the time of data collection, requests for TA exceeded the center’s capacity. 

NOTE: IRIS-II was funded through 2011; information in the table reflects center activities at the time of data collection. 

SOURCE: Interview with center director, conducted in January 2010 (appendix A), and inventory of products and services 
collected from center director between November 2009 and March 2010 (appendix B). 

In addition to other products and services, IRIS-II produced web-based interactive 
training modules, which accounted for 40 percent of costs for product development and 
dissemination at the time of data collection (table 4.23). 
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Table 4.23: Count of IDEA and Research for Inclusive Settings Center for Training 
Enhancements (IRIS-II) products and services and their reported costs 


Product type 

Number of 
products 

Reported costs 

IRIS-II products/services 

90 

$2,193,000 

Conferences, institutes, or workshops 

13 1 

570,000 

Materials supporting universal technical assistance (TA) 

2 2 

44,000 

Presentations or webinars 

65 3 

443,000 

Web-based training modules, including supporting 
materials 

8 4 

880,000 

Websites, including activities promoting website use 

l 5 

217,000 

Other products or services 

l 6 

38,000 


TABLE READS: IRIS-II identified 90 products or services developed with PDP funding at the time of data collection and 
reported costs of $2. 193 million for product development and dissemination. 

1 One IRIS Institute and 12 seminars to help faculty incorporate IRIS materials into syllabi and department curricula. 

2 Three case study units providing practice with school-based scenarios counted as one product; 10 postsecondary lesson 
activities counted as one product. 

3 Individual conference presentations and workshops, webinars, and TA and outreach sessions for special education faculty 
and professional development providers. 

4 Web-based interactive training modules based on the Star Legacy Cycle’s 5 stages: The Challenge, Thoughts, 
Perspectives and Resources, Assessment, and Wrap-Up. 

5 One website, including phone-based guided tours of the website to familiarize users with the content and navigation, with 
access to center products. 

6 Promotional materials. 

NOTE: IRIS-II was funded through 2011; products and services in this table include only those developed by the time of 
data collection. Costs have been rounded to the nearest thousand. 

SOURCE: Inventory of products and services collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed 10 products sampled from 
the inventory of products provided by IRIS-II. Panelists reviewed each selected IRIS-II 
product/service based on four quality indicators and five relevance/usefulness indicators, with 
rating values ranging from 1 (very low) to 5 (very high). Not all indicators were relevant to all 
products. 

For the two reviewed products identified by center staff as signature works, all quality 
ratings and 93 percent of the relevance/usefulness ratings were high or very high (figure 4.8). 
Across the eight nonsignature products, 82 percent of the quality ratings and 90 percent of the 
relevance/usefulness ratings were high or very high. The mean ratings for signature products 
were 4.75 for quality and 4.47 for relevance/usefulness, and the mean ratings for nonsignature 
products were 4.33 for quality and 4.44 for relevance/usefulness (table 4.24). Table 4.24 
provides additional details regarding the number of indicators, number of ratings, and scores for 
the specific types of products that were reviewed. 
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Figure 4.8: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature products/ 
services of the IDEA and Research for Inclusive Settings Center for Training 
Enhancements (IRIS-II) 


Percent 


100 
80 
60 
40 
20 
0 

Quality Relevance/ Quality Relevance/ 

Usefulness Usefulness 


1 



Very Low 
Low 
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(n=2 products/services, with 24 quality 
ratings and 30 relevance/usefulness ratings) 


Nonsignature products/services 
(n=8 products/services, with 96 quality 
ratings and 114 relevance/usefulness ratings) 


FIGURE READS: For the two signature IRIS-II products reviewed by an expert panel, none of the 24 quality ratings were at 
the level of 1 (very low), 2 (low), or 3 (moderate); 25 percent were 4 (high), and 75 percent were 5 (very high). 

1 Center staff members were asked to identify up to 10 percent of the center’s products/services as signature works. Signature 
works were products or services that (1) were prominent in terms of the proportion of the center’s overall effort, (2) were 
exclusively or nearly exclusively a product of the center’s effort, (3) were predominately supported by grant funds or unlikely to 
have occurred without grant support, and (4) made a consequential contribution to the field or to the class of customers served. 
Neither websites nor products categorized as other were included in the expert review process. 

NOTE: Panelists reviewed each selected IRIS-II product/service based on four quality indicators and five relevance/usefulness 
indicators. In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The 
number of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular 
product. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.24: Expert ratings on sampled signature and nonsignature IDEA and Research for 
Inclusive Settings Center for Training Enhancements (IRIS-II) products/ 
services, by review dimension and type of product/service 






Quality 


Relevance/usefulness 

IRIS-II products/services 
reviewed 

Number of 

products/services 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Number of 
indicators 

Number of 
ratings 

Mean 

SD 

Signature 


2 

4 

24 

4.75 

0.44 

5 

30 

4.47 

0.63 

Web-based training modules, 











including supporting 
materials 


2 

4 

24 

4.75 

0.44 

5 

30 

4.47 

0.63 

Nonsignature 


8 

4 

96 

4.33 

0.76 

5 

114 

4.43 

0.69 

Materials supporting 
universal technical assistance 


2 

4 

24 

4.00 

0.83 

5 

29 

4.28 

0.75 

Presentations or webinars 


1 

4 

12 

4.42 

0.51 

5 

10 

4.00 

0.47 

Web-based training modules, 











including supporting 
materials 


5 

4 

60 

4.45 

0.75 

5 

75 

4.56 

0.66 


TABLE READS: Two IRIS-II signature products were reviewed. Each of three experts rated up to four quality indicators for 
each product, resulting in 24 ratings. The mean rating for quality was 4.75 ( sd=0.44 ). The experts rated up to five relevance/ 
usefulness indicators for each product, resulting in 30 ratings. The mean rating for relevance/usefulness was 4.47 (sd=0.63). 


NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
IRIS-II, experts made 264 ratings across the 10 products/services. All ratings were on a 5-point scale, with the following values: 
l=very low, 2=low, 3=moderate, 4=high, 5=very high. Neither websites nor products categorized as other were included in the 
expert review process. 

SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 
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National Institute for Urban School Improvement Principal Leadership Academies 
Initiative 

NIUSI-PLAN LeadScape was designed to assist school principals in developing and 
implementing inclusive schools to ensure that students with and without disabilities meet or 
exceed state academic standards. The center was funded 2006-11 and housed at the University of 
Arizona. The center conducted face-to-face and online trainings, provided on-site coaching, and 
disseminated materials to support the use of inclusive practices, (table 4.25). 


Table 4.25: Descriptive information reported by the National Institute for Urban School 
Improvement Principal Leadership Academies Initiative (NIUSI-PLAN 
LeadScape) 


Area 

Center-specific description 

Primary center 
strategies 

• Developing, testing, and validating a combination of face-to-face instruction, 
online learning, and on-site coaching with a pilot group of principals, 
followed by the launch of full-scale implementation with an additional 400 
principals; 

• Holding annual 3-day institutes with participating principals; 

• Providing online professional learning modules and dialogue guides; 

• Providing on-site coaching; 

• Developing and supporting a website with professional learning activities, 
products, and resources; and 

• Disseminating materials to local education agencies about website activities, 
products, and resources. 

Most frequently 
used products or 
services 

• Website access to professional learning activities, products, and resources. 

Link of center 
activities to prior 
work in the field 

• The center was a continuation of work done by the National Center for 
Culturally Responsive Educational Systems and the National Center for 
Urban School Improvement. Many of the products produced through the 
other centers were offered to principals through NIUSI-PLAN LeadScape. 

Challenges faced 
by center 

• Setting up and using data from school districts to accommodate the many 
different data platforms used in districts; and 

• Helping schools that have access to data understand how this platform can 
enhance their work. 


NOTE: NIUSI-PLAN LeadScape was funded through 2011; information in the table reflects center activities at the time of 
data collection. 

SOURCE: Interview with center director, conducted in January 2010 (appendix A). 


Among other products and services, NIUSI-PLAN LeadScape provided a series of onsite 
institutes for principals focused on inclusive schools that accounted for 49 percent of costs for 
product development and dissemination at the time of data collection (table 4.26). 
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Table 4.26: Count of National Institute for Urban School Improvement Principal 

Leadership Academies Initiative (NIUSI-PLAN LeadScape) products and 
services and their reported costs 


Product type 

Number of 
products 

Reported costs 

NIUSI-PLAN LeadScape products and services 

25 

$ 419,000 

Briefs or short reports 

6 1 

$23,000 

Conferences, institutes, or workshops 

5 2 

205,000 

Materials supporting universal technical assistance 

l 3 

5,000 

Web-based training modules, including supporting 
materials 

12 4 

36,000 

Websites, including activities promoting website use 

l 5 

150,000 


TABLE READS: NIUSI-PLAN LeadScape identified 25 products or services developed with PDP funding at the time of 
data collection and reported costs of $419,000 for product development and dissemination. 

*Six topical briefs designed to provide practitioners with research-based guidelines for inclusive practices. 

2 Three-day institutes for participating school principals. 

3 Framework for using culturally responsive cognitive coaching in schools. 

4 Twelve professional learning modules on culturally responsive practices, early intervening strategies, and universal design 
for learning approaches. 

5 Website with individual school wikis, research-based briefs and journal abstracts related to inclusive practices, 
downloadable professional learning and school improvement resources, expert blog on inclusive practices, school-level data, 
and online discussion groups and shared files. 

NOTE: NIUSI-PLAN LeadScape was funded through 2011; products and services in this table include only those developed 
by the time of data collection. 

SOURCE: Inventory of products and services collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed 10 products sampled from 
the inventory of products provided by NIUSI-PLAN LeadScape. Panelists rated each sampled 
NIUSI-PLAN LeadScape product/service based on nine quality indicators and four 
relevance/usefulness indicators, with rating values ranging from 1 (very low) to 5 (very high). 
Not all indicators were relevant to all products. 

For the one reviewed product identified by center staff as a signature work, 63 percent of 
the quality ratings and 60 percent of the relevance/usefulness ratings were high or very high 
(figure 4.9). Across the nine nonsignature products, 60 percent of the quality ratings and 52 
percent of the relevance/usefulness ratings were high or very high. The mean ratings for the 
signature product were 3.93 for quality and 3.90 for relevance/usefulness, and the mean ratings 
for nonsignature products were 3.87 for quality and 3.68 for relevance/usefulness (table 4.27). 
Table 4.27 provides additional details regarding the number of indicators, number of ratings, and 
scores for the specific types of products that were reviewed. 
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Figure 4.9: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature products/ 
services of the National Institute for Urban School Improvement Principal 
Leadership Academies Initiative (NIUSI-PLAN LeadScape) 

Percent 



Quality 


Relevance/ 

Usefulness 


Quality 


Relevance/ 

Usefulness 


Very Low 
Low 

■ Moderate 

■ High 

■ Very High 


Signature products/services 1 
(n=l product/service, with 27 quality 
ratings and 10 relevance/usefulness ratings) 


Nonsignature products/services 
(n=9 products/services, with 221 quality 
ratings and 97 relevance/usefulness ratings) 


FIGURE READS: For the signature NIUSI-PLAN LeadScape product/service reviewed by an expert panel, none of the 27 
quality ratings were at the level of 1 (very low), 4 percent were 2 (low), 33 percent were 3 (moderate), 30 percent were 4 
(high), and 33 percent were 5 (very high). 

1 Center staff members were asked to identify up to 10 percent of the center’s products/services as signature works. Signature 
works were products or services that (1) were prominent in terms of the proportion of the center’s overall effort, (2) were 
exclusively or nearly exclusively a product of the center’s effort, (3) were predominately supported by grant funds or unlikely to 
have occurred without grant support, and (4) made a consequential contribution to the field or to the class of customers served. 
Center websites were not included in the expert panel review. 

NOTE: Panelists reviewed each selected NIUSI-PLAN product/service based on nine quality indicators and four relevance/ 
usefulness indicators. In general, the number of ratings is equal to three raters x the number of products x the number of 
indicators. The number of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” 
to a particular product. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.27: Expert ratings on sampled National Institute for Urban School Improvement 
Principal Leadership Academies Initiative (NIUSI-PLAN LeadScape) 
products/services, by review dimension and product signature status 
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Mean 


SD 


Signature 

1 

9 

27 

3.93 

0.92 

4 

10 

3.90 

1.10 

Materials supporting 
universal technical assistance 

1 

9 

27 

3.93 

0.92 

4 

10 

3.90 

1.10 


Nonsignature 

9 

9 

221 

3.87 

0.88 

4 

97 

3.68 

1.00 

Briefs or short reports 

2 

9 

43 

3.56 

0.83 

4 

21 

3.43 

0.98 

Conferences, institutes, or 



52 

3.85 

0.80 


20 

3.75 

1.02 

workshops 

Web-based training modules, 

2 

9 

4 

including supporting 
materials 

5 

9 

126 

3.99 

0.90 

4 

56 

3.75 

1.00 


TABLE READS: One NIUSI-PLAN LeadScape signature product was reviewed. Each of three experts rated up to nine quality 
indicators for each product, resulting in 27 ratings. The mean rating for quality was 3.93 ( sd=0.92 ). The experts rated up to four 
relevance/usefulness indicators for each product, resulting in 10 ratings. The mean rating for relevance/usefulness was 3.90 
(sd=1.10). 


NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
NIUSI-PLAN LeadScape, experts made 355 ratings across the 10 products/services. All ratings were on a 5-point scale, with the 
following values: l=very low, 2=low, 3=moderate, 4=high, 5=very high. Center websites were not included in the expert review 
process. 

SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 
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National Professional Development Center on Inclusion 

NPDCI was designed to increase the number of high-quality early childhood personnel 
serving children in inclusive preschool settings by improving the quality and accessibility of PD, 
training, and TA. The center was funded 2006-11 and housed at the University of North Carolina 
at Chapel Hill’s Frank Porter Graham Child Development Institute. The center provided 
rigorous, ongoing training and technical assistance to select states, and produced additional 
resources available to all states (table 4.28). 


Table 4.28: Descriptive information reported by the National Professional Development 
Center on Inclusion (NPDCI) 


Area 

Center-specific description 

Primary center 
strategies 

• Working with selected states to identify professional development (PD) 
needs and resources and provide rigorous, ongoing PD, training, and 
technical assistance (TA) to develop statewide systems of PD for supporting 
inclusion in early childhood; 

• Developing and disseminating resources to assist all states in increasing the 
number of high-quality personnel serving preschoolers with disabilities and 
their families in inclusive settings, including concept papers, research 
syntheses, planning and facilitation tools, and webinars; 

• Conducting conference presentations and intensive workshops; and 

• Developing and supporting an interactive website with tools, presentations, 
multimedia, and password-protected space for state work sites. 

Most frequently 
used products or 
services 

• NPDCI website with workspace for participating states and a variety of 
resources for any state-level early childhood personnel; 

• Concept papers related to PD of early childhood personnel; and 

• A position paper on early childhood inclusion. 

Challenges faced 
by center 

• Turnover among some of the state liaisons due to illness, retirement, or 
resignation; 

• Changes in state leadership leading to confusion about state priorities and 
inconsistent funding for PD initiatives and cross-sector work; 

• Moving states from cross-sector co-existence toward cross-sector integration; 
and 

• The necessity for states to respond to changes at the federal level as well as 
impact of the economic downturn. 

NOTE: NPDCI was funded through 2011; information in the table reflects center activities at the time of data collection. 
There was not sufficient infonnation available to report on the link of center activities to prior work in the field. 

SOURCE: Interview with center director, conducted in January 2010 (appendix A), and Inventory of products and services, 
collected from center director between November 2009 and March 2010 (appendix B). 


In addition to other products and services at the time of data collection, NPDCI had 
worked with eight states to develop statewide comprehensive systems of high-quality, cross- 
sector PD to support inclusion, which accounted for 53 percent of costs for product development 
and dissemination (table 4.29). 
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Table 4.29: Count of National Professional Development Center on Inclusion (NPDCI) 
products and services and their reported costs 


Product type 

Number of products 

Reported costs 

NPDCI products and services 

119 

$1,173,000 

Briefs or short reports 

4 1 

$85,000 

Conferences, institutes, or workshops 

17 2 

36,000 

Intensive, focused technical assistance (TA) 

8 3 

625,000 

Materials supporting universal TA 

2 4 

34,000 

Presentations or webinars 

71 5 

140,000 

Research papers, articles, dissertations, or book chapters 

15 6 

64,000 

Websites, including activities promoting website use 

l 7 

185,000 

Other products or services 

l 8 

2,000 


TABLE READS: NPDCI identified 119 products and services developed with PDP funding at the time of data collection 
and reported costs of $ 1 . 1 73 million for product development and dissemination. 

1 Two concept papers, a research synthesis on early childhood inclusion, and research findings on professional development 
(PD) in early childhood. 

2 Eleven seminars and six workshops. 

3 Work with eight states to develop statewide comprehensive systems of high-quality, cross-sector PD to support inclusion. 

4 Planning and facilitation tools for PD; and a survey designed to gather information about cross-sector, statewide PD in 
early childhood. 

5 Seven webinars and 64 presentations. 

6 Six book chapters, seven journal articles, ajoint position statement of the Division of Early Childhood of the Council for 
Exceptional Children and the National Association for the Education of Young Children, and a journal special issue. 

7 Website with password-protected worksites for participating states, as well as a variety of resources targeted to any state- 
level early childhood personnel. 

s Promotional materials. 

NOTE: NPDCI was funded through 2011; products and services in this table include only those developed by the time of 
data collection. Cost data have been rounded to the nearest thousand. 

SOURCE: Inventory of products and services, collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed 10 products from the 
inventory of products provided by NPDCI. Panelists reviewed each selected NPDCI product 
based on eight quality indicators and six relevance/usefulness indicators, with rating values 
ranging from 1 (very low) to 5 (very high). Not all indicators were relevant to all products. 

For the three reviewed products identified by center staff as signature works, 74 percent 
of the quality ratings and 7 1 percent of the relevance/usefulness ratings were high or very high 
(figure 4.10). Across the seven nonsignature products, 48 percent of quality ratings and 49 
percent of relevance/usefulness ratings were high or very high. The mean ratings for signature 
products were 3.87 for quality and 3.84 for relevance/usefulness, and the mean ratings for 
nonsignature products were 3.42 for quality and 3.30 for relevance/usefulness (table 4.30). Table 
4.30 provides additional details regarding the number of indicators, number of ratings, and scores 
for the specific types of products that were reviewed. 
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Figure 4.10: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature 
products/services of the National Professional Development Center on 
Inclusion (NPDCI) 


Percent 



Usefulness Usefulness 


Very Low 
Low 

■ Moderate 

■ High 

■ Very High 


Signature products/services 1 
(n=3 products/services, with 68 quality 
ratings and 51 relevance/usefulness ratings) 


Nonsignature produets/services 
(n=7 products/services, with 138 quality 
ratings and 116 relevance/usefulness ratings) 


FIGURE READS: For the three signature products of NPDCI reviewed by an expert panel, 3 percent of the 68 quality ratings 
were at the level of 1 (very low), 1 percent were 2 (low), 22 percent were 3 (moderate), 53 percent were 4 (high) and 21 percent 
were 5 (very high). 

1 Center staff members were asked to identify up to 10 percent of the center’s products/services as signature works. Signature 
works were products or services that (1) were prominent in terms of the proportion of the center’s overall effort, (2) were 
exclusively or nearly exclusively a product of the center’s effort, (3) were predominately supported by grant funds or unlikely 
to have occurred without grant support, and (4) made a consequential contribution to the field or to the class of customers 
served. 

NOTE: Panelists reviewed each selected NPDCI product/service based on eight quality indicators and six relevance/usefulness 
indicators. In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The 
number of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular 
product. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.30: Expert ratings on sampled National Professional Development Center on 
Inclusion (NPDCI) products/services, by review dimension and product 
signature status 





Quality 


Relevance/usefulness 

NPDCI products/services 
reviewed 

Number of 
products/services 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Signature 

3 

8 

68 

3.87 

0.86 

6 

51 

3.84 

0.83 

Intensive, focused technical 

1 


21 

3.19 

0.75 


16 

3.75 

0.58 

assistance (TA) 

8 

6 

Research papers, articles, 










dissertations, or book 
chapters 

2 

8 

47 

4.17 

0.73 

6 

35 

3.89 

0.93 

Nonsignature 

7 

8 

138 

3.42 

0.78 

6 

116 

3.30 

1.02 

Intensive, focused TA 

1 

8 

23 

3.35 

0.71 

6 

17 

3.53 

1.01 

Presentations or webinars 

5 

8 

95 

3.52 

0.77 

6 

81 

3.37 

0.95 

Research papers, articles, 










dissertations, or book 
chapters 

1 

8 

20 

3.05 

0.83 

6 

18 

2.78 

1.22 


TABLE READS: Three NPDCI signature products were reviewed. Each of three experts rated up to eight quality indicators for 
each product, resulting in 68 ratings. The mean rating for quality was 3.87 ( sd=0.86 ). The experts rated up to six 
relevance/usefulness indicators for each product, resulting in 51 ratings. The mean rating for relevance/usefulness was 3.84 
(sd=0.83). 

NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
NPDCI, experts made 373 ratings across the 10 products/services. Five ratings were not included in the analysis for the reason 
stated above (f). All ratings were on a 5-point scale, with the following values: l=very low, 2=low, 3=moderate, 4=high, 5=very 
high. Neither center websites nor products categorized as other were included in the expert review process. 

SOURCE: Rubric -based reviews by expert panel, conducted May through September 2010. 
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The Center to Mobilize Early Childhood Knowledge 

CONNECT was designed to assist faculty and PD providers to prepare highly qualified 
personnel in special education, related services, early intervention, and early care and education 
to work with young children with disabilities and their families. The center is housed at the 
University of North Carolina at Chapel Hill’s Frank Porter Graham Child Development Institute. 
PDP funding for the center began in 2007 and will continue through 2012. The center has 
developed resources for early childhood practitioners, faculty, and PD providers around 
evidence-based approaches in early intervention/early childhood special education (table 4.31). 

Table 4.31: Descriptive information reported by the Center to Mobilize Early Childhood 
Knowledge (CONNECT) 


Area Center-specific description 

Primary center • Increasing the competence of professional development (PD) providers for 

strategies designing, delivering, and evaluating evidence-based approaches in early 

intervention/early childhood special education; 

• Developing evidence-based content modules on teaching and intervention 
practices in early childhood education for children with special needs; 

• Developing pedagogical modules on research-based or promising PD 
approaches and models in early childhood education for children with special 
needs; 

• Creating mechanisms for accessing materials, such as a searchable database 
of downloadable resources, online interactive modules and materials, and 
syntheses of research on teaching and intervention practices; and 

• Providing targeted technical assistance to faculty and PD providers. 

• Finding a balance between content breadth and depth in development of the 
first module; 

• Initial lack of necessary technology infrastructure and expertise, which was 
remedied through use of a new web platform and the addition of qualified 
staff; and 

• Hiring new staff and managing evolving staff roles and responsibilities. 

NOTE: CONNECT was funded through 2012; information in the table reflects center activities at the time of data 
collection. There was not suffcient information available to report the most frequently used products or services or the link 
of center activities to prior work in the field. 

SOURCE: Interview with center director, conducted in January 2010 (appendix A), and Online Discretionary Grants Public 
Database accessed through the Office of Special Education Programs Technical Assistance and Dissemination Network. 


Challenges faced 
by center 


In addition to other products and services at the time of data collection, CONNECT had 
developed the first in a series of web-based instructional modules with supporting materials, 
which accounted for 65 percent of reported costs for product development and dissemination 
(table 4.32). 
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Table 4.32: Count of Center to Mobilize Early Childhood Knowledge (CONNECT) 
products and services and their reported costs 


Product type 

Number of products 

Reported costs 

CONNECT products and services 

18 

$777,000 

Conferences, institutes, or workshops 

2 1 

$9,000 

Presentations or webinars 

9 2 

27,000 

Research papers, articles, dissertations, or book chapters 

2 3 

1,000 

Web-based training modules, including supporting 
materials 

3 4 

506,000 

Websites, including activities promoting website use 

l 5 

234,000 

Other products or services 

l 6 

400 


TABLE READS: CONNECT identified 18 products and services developed with PDP funding at the time of data collection 
and reported costs of $777,000 for product development and dissemination. 

1 Two day-long workshops for faculty and professional development (PD) providers. 

2 National webinars, web-based PD resources, and presentations. 

3 One book chapter and one article on evidence-based practices for supporting inclusion. 

4 One interactive web-based module for early childhood practitioners, faculty, and PD providers; one video about the 
CONNECT web-based modules; and one multi-media module on early childhood inclusion. 

5 Website for online collaborative work with a searchable database of planning tools, articles, concept papers, research 
summaries, publications, presentations, activities, a newsletter and blogs for parents, policymakers, practitioners, and 
providers. 

6 Promotional materials. 

NOTE: CONNECT was funded through 2012; products and services in this table include only those developed by the time 
of data collection. Cost data have been rounded to the nearest thousand. 

SOURCE: Inventory of products and services, collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed five products from the 
inventory of products provided by CONNECT. Panelists reviewed each selected CONNECT 
product/service based on eight quality indicators and six relevance/usefulness indicators, with 
rating values ranging from 1 (very low) to 5 (very high). Not all indicators were relevant to all 
products. 

For the one sampled product identified by center staff as a signature works, 92 percent of 
the quality ratings and 95 percent of the relevance/usefulness ratings were high or very high 
(figure 4. 1 1). Across the four nonsignature products, 83 percent of quality ratings and 72 percent 
of relevance/usefulness ratings were high or very high. The mean ratings for the signature 
product were 4.46 for quality and 4.65 for relevance/usefulness, and the mean ratings for the 
nonsignature products were 4.25 for quality and 4.20 for relevance/usefulness (table 4.33). Table 
4.33 provides additional details regarding the number of indicators, number of ratings, and scores 
for the specific types of products that were reviewed. 
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Figure 4.11: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature products/ 
services of the Center to Mobilize Early Childhood Knowledge (CONNECT) 


Percent 



Usefulness Usefulness 


Very Low 
Low 

■ Moderate 

■ High 

■ Very High 


Signature products/services 1 
(n=l product/service, with 24 quality 
ratings and 17 relevance/usefulness ratings) 


Nonsignature products/services 
(n=4 products/services, with 72 quality 
ratings and 64 relevance/usefulness ratings) 


FIGURE READS: For the signature CONNECT product reviewed by an expert panel, none of the 24 quality ratings were at the 
level of 1 (very low) or 2 (low), 8 percent were 3 (moderate), 38 percent were 4 (high), and 54 percent were 5 (very high). 

1 Center staff members were asked to identify up to 10 percent of the center’s products/services as signature works. Signature 
works were products or services that (1) were prominent in terms of the proportion of the center’s overall effort, (2) were 
exclusively or nearly exclusively a product of the center’s effort, (3) were predominately supported by grant funds or unlikely to 
have occurred without grant support, and (4) made a consequential contribution to the field or to the class of customers served. 
NOTE: Panelists reviewed each selected CONNECT product/service based on eight quality indicators and six 
relevance/usefulness indicators. In general, the number of ratings is equal to three raters x the number of products x the 
number of indicators. The number of ratings was reduced, however, when a rater considered a particular indicator to be “not 
applicable” to a particular product. Detail may not sum to totals due to rounding. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.33: Expert ratings on sampled Center to Mobilize Early Childhood Knowledge 
(CONNECT) products/services, by review dimension and product signature 
status 





Quality 


Relevance/usefulness 

CONNECT 

products/services reviewed 

Number of 
products/services 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Number of 
indicators 

Number of 
ratings 

Mean 

SD 

Signature 

1 

8 

24 

4.46 

0.66 

6 

17 

4.65 

0.61 

Web-based training modules, 










including supporting 
materials 

1 

8 

24 

4.46 

0.66 

6 

17 

4.65 

0.61 

Nonsignature 

4 

8 

72 

4.25 

0.80 

6 

64 

4.20 

0.98 

Presentations or webinars 

1 

8 

20 

4.00 

0.86 

6 

14 

4.29 

0.91 

Research papers, articles, 










dissertations, or book 
chapters 

Web-based training modules, 

2 

6 

33 

4.09 

0.80 

6 

32 

3.91 

1.06 

including supporting 
materials 

1 

8 

19 

4.79 

0.42 

6 

18 

4.67 

0.69 


TABLE READS: One CONNECT signature product was reviewed. Each of three experts rated up to eight quality indicators for 
each product, resulting in 24 ratings. The mean rating for quality was 4.46 ( sd=0.66 ). The experts rated up to six relevance/ 
usefulness indicators for each product, resulting in 17 ratings. The mean rating for relevance/usefulness was 4.65 (sd=0.61). 
NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
CONNECT, experts made 177 ratings across the 5 products/services. All ratings were on a 5-point scale, with the following 
values: l=very low, 2=low, 3=moderate, 4=high, 5=very high. Neither websites nor products categorized as other were included 
in the expert review process. 

SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 
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National Center to Inform Policy and Practice in Special Education Professional 
Development 

NCIPP, housed at the University of Florida, was designed to recommend policies and 
practices to improve teacher induction and mentoring, collaboration, and preservice preparation. 
PDP funding for the center began in 2007 and will continue through 2012. NCIPP has produced 
research syntheses, policy analyses, and information briefs targeted to college faculty, as well as 
state, local, and building level administrators, (table 4.34). 

Table 4.34: Descriptive information reported by the National Center to Inform Policy and 
Practice in Special Education Professional Development (NCIPP) 


Area 

Primary center 
strategies 


Most frequently 
used products or 
services 

Link of center 
activities to prior 
work in the field 


Challenges faced 
by center 


Center-specific description 

• Synthesizing, publishing, and disseminating research on induction and 
mentoring, collaboration, and preservice preparation, through research 
syntheses, policy analyses, and information briefs; 

• Identifying and recommending to institutions of higher education (IHEs), 
state education agencies, and local education agencies 

a. policies and practices in special education teacher preparation that 
improve retention; and 

b. implementation strategies that provide beginning special education and 
regular education teachers with the knowledge and skills to effectively 
support students with disabilities in different classroom settings, with a 
particular focus on urban and rural schools. 

• Developing model cases that describe well-established programs for 
enhanced induction and mentoring for special educators; and 

• Providing targeted technical assistance (TA) to states and intensive TA to 

districts and IHEs. 

• Not yet known at the time of data collection. 


• NCIPP is not a continuation of any previous work, though many of the 

individuals involved in NCIPP worked together on a prior Office of Special 
Education Programs project, the Center on Personnel Studies in Special 
Education. 

• Inadequate research was available on evidence based practices at the 
initiation of the grant. 


NOTE: NCIPP was funded through 2012; information in the table reflects center activities at the time of data collection. 
SOURCE: Interview with center director conducted in December 2009 (appendix A); and NCIPP website 
(http://ncipp.education.ufl.edu). 


In addition to other products and services at the time of data collection, NCIPP had 
developed several research papers, which accounted for 61 percent of costs for product 
development and dissemination (table 4.35). 
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Table 4.35: Count of National Center to Inform Policy and Practice in Special Education 
Professional Development (NCIPP) products and services and their reported 
costs 


Product type 

Number of 
products 

Reported costs 

NCIPP products and services 

39 

$262,000 

Briefs or short reports 

29 1 

$93,000 

Presentations or webinars 

6 2 

9,000 

Research papers, articles, dissertations, or book chapters 

4 3 

160,000 


TABLE READS: NCIPP identified 39 products and services developed with PDP funding at the time of data collection and 
reported costs of $262,000 for product development and dissemination. 

1 Information briefs targeted to state and local education agencies and building administrators. 

2 Presentations delivered at national conferences and available for the public via podcasts. 

3 Research syntheses targeted to institutions of higher education faculty. 

NOTE: NCIPP was still active at the time of data collection; thus, the counts of products and services provided in this table 
are incomplete. Costs have been rounded to the nearest thousand. The NCIPP website was not included in the count of 
products and services provided by the center director. 

SOURCE: Inventory of products and services, collected from center director between November 2009 and March 2010 
(appendix B). 


In the summer of 2010, three expert panel members reviewed five products from the 
inventory of products provided by NCIPP. Panelists reviewed each selected NCIPP 
product/service based on six quality indicators and four relevance/usefulness indicators, with 
rating values ranging from 1 (very low) to 5 (very high). Not all indicators were relevant to all 
products. 

For the one reviewed product identified by center staff as a signature work, 83 percent of 
the quality ratings and 92 percent of the relevance/usefulness ratings were high or very high 
(figure 4. 12). Across the four nonsignature products, 88 percent of quality ratings and 90 percent 
of relevance/usefulness ratings were high or very high. The mean ratings for the signature 
product were 4.50 for quality and 4.67 for relevance/usefulness, and the mean ratings for the 
nonsignature products were 4.59 for quality and 4.54 for relevance/usefulness (table 4.36). Table 
4.36 provides additional details regarding the number of indicators, number of ratings, and scores 
for the specific types of products that were reviewed. 
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Figure 4.12: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 
highest) of quality and relevance for signature and nonsignature 
products/services of the National Center to Inform Policy and Practice in 
Special Education Professional Development (NCIPP) 

Percent 



Usefulness Usefulness 


Very Low 
Low 

■ Moderate 

■ High 

■ Very High 


Signature products/services 1 
(n=l product/service, with 14 quality 
ratings and 12 relevance/usefulness ratings) 


Nonsignature products/services 
(n=4 products/services, with 71 quality 
ratings and 48 relevance/usefulness ratings) 


FIGURE READS: For the sampled signature NCIPP product reviewed by an expert panel, none of the 14 quality ratings were 
rated 1 (very low) or 2 (low), 17 percent or quality ratings were 3 (moderate), 25 percent were 4 (high), and 58 percent were 5 
(very high). 

1 Center staff members were asked to identify up to 10 percent of the center’s products/services as signature works. Signature 
works were products or services that (1) were prominent in terms of the proportion of the center’s overall effort, (2) were 
exclusively or nearly exclusively a product of the center’s effort, (3) were predominately supported by grant funds or unlikely to 
have occurred without grant support, and (4) made a consequential contribution to the field or to the class of customers served. 

NOTE: Panelists reviewed each selected NCIPP product/service based on six quality indicators and four relevance/usefulness 
indicators. In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The 
number of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular 
product. Detail may not sum to totals due to rounding. 

SOURCE: Expert panel reviews, conducted May through September 2010 using the rubric in appendix C. 
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Table 4.36: Expert ratings on sampled National Center to Inform Policy and Practice in 
Special Education Professional Development (NCIPP) products/services, by 
review dimension and product signature status 





Quality 


Relevance/usefulness 

NCIPP products/services 
reviewed 

Number of 
products/services 

Number of 
indicators rated 

Number of 
ratings 

Mean 

SD 

Number of 
indicators 

Number of 
ratings 

Mean 

SD 

Signature 

1 

6 

14 

4.50 

0.76 

4 

12 

4.67 

0.65 

Briefs or short reports 1 

1 

6 

14 

4.50 

0.76 

4 

12 

4.67 

0.65 

Nonsignature 

Research papers, articles, 

4 

6 

71 

4.59 

0.69 

4 

48 

4.57 

0.66 

dissertations, or book 
chapters 

4 

6 

71 

4.59 

0.69 

4 

48 

4.54 

0.68 


TABLE READS: One NCIPP signature product was reviewed. Each of three experts rated up to six quality indicators for each 
product, resulting in 14 ratings. The mean rating for quality was 4.50 ( sd=0 . 76). The experts rated up to four 
relevance/usefulness indicators for each product, resulting in 12 ratings. The mean rating for relevance/usefulness was 4.67 
(sd=0.65). 

1 The signature product sampled from NCIPP consisted of 29 individual briefs reviewed by the expert panel as a single product. 


NOTE: In general, the number of ratings is equal to three raters x the number of products x the number of indicators. The number 
of ratings was reduced, however, when a rater considered a particular indicator to be “not applicable” to a particular product. For 
NCIPP, experts made 145 ratings across the 5 products/services. All ratings were on a 5-point scale, with the following values: 
l=very low, 2=low, 3=moderate, 4=high, 5=very high. 


SOURCE: Rubric-based reviews by expert panel, conducted May through September 2010. 
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5. Design for the Substudy of PDP Training Grant Applicants 


The substudy of PDP training grant applicants was intended to (1) describe courses of 
study that were the focus of applications in FY 2006 and FY 2007, (2) document changes to 
courses of study since application, and (3) assess the quality and relevance/usefulness of a 
sample of components within funded courses of study that were developed or significantly 
modified during the funding period. Data were collected from fall 2009 through fall 2010. Data 
sources included (1) relevant extant data, (2) a web-based survey of funded and nonfunded 
applicants, and (3) an expert review of sampled components within funded courses of study. 


The Unit of Analysis: Course of Study 

The primary unit of analysis for the substudy of PDP training grant applicants was the 
course of study. We defined a course of study as follows: A set of courses to prepare candidates 
to perform a particular professional or paraprofessional role; this set of courses almost always 
results in a degree, a particular credential ( i. e. , license, certificate, or endorsement), or both. 

The setting of courses is another defining aspect of the course of study (e.g., on a particular 
campus, online). 

The substudy of PDP training grant applicants focused on the course or courses of study 
that were at the center of each training grant application. IHEs apply to a particular PDP 
priority/focus area based on the area of preparation on which their course of study is or will be 
focused, including the degree level for which candidates will be prepared. IHEs may submit 
multiple applications in a given year within or across priority/focus areas for funding related to 
different courses of study. A single PDP application may include one or more courses of study. 
The majority of analyses in this study were conducted at the course of study level. 

To illustrate, figure 5.1 presents an example of four courses of study from a single 
university, contained within three applications, all submitted by the College of Education. One 
application originated from the Department of Counseling and Educational Psychology and 
focused on a single course of study — Ph.D. in school psychology. A second application came 
from the Department of Special Education and Communications Disorders and focused on a 
single course of study — Ed.S. in special education administration. The third application, also 
from the Department of Special Education and Communications Disorders, included two courses 
of study — master’s in communication sciences and disorders, and a credential-only program in 
speech-language pathology. 
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Figure 5.1: Four courses of study included in the substudy of PDP training grant 
applicants: Three applications submitted by University A’s College of 
Education Departments of Counseling and Educational Psychology and 
Special Education and Communication Disorders 



Research questions for this substudy are descriptive and are based on the following 
conception of the PDP training grant competitions: 

• IHEs with and without PDP funding have courses of study that prepare candidates to 
serve children with disabilities. 

• IHEs apply to the PDP for funding to maintain or enhance an existing course of study or 
to develop a course of study that is substantially different from any existing course of 
study. 

• Courses of study that are funded may attract and support more candidates, add or 
significantly modify components, reorganize, or develop as a substantially different 
course of study. 

• Courses of study that are not funded may do any of the above without the benefit of PDP 
funding. Or, they may attract and support fewer candidates, discontinue or degrade 
components, cease to exist, or not develop. 

Table 5.1 lists the primary research questions for the substudy of PDP training grant applicants, 
along with an overview of the data sources and analyses associated with each question. 
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Table 5.1: Research questions, data sources, and analyses for the substudy of PDP training 
grant applicants 


Research question 

Data sources 

Analyses 

1 . What were the characteristics of 
funded courses of study at 
institutions of higher education 
(IHEs) awarded PDP training 
grants in FY 2006 or FY 2007? 

Review of funded applications, 
U.S. Department of Education 
Grant Award Database, Office 
of Special Education Programs 
Discretionary Grants Database, 
IHE Course of Study Survey. 

Counts, percentages, and 
means, disaggregated by PDP 
priority/focus area. 

2. How did funded courses of 
study use PDP training grant 
funding? 

IHE Course of Study Survey 

Counts, percentages, and 
means, disaggregated by PDP 
priority/focus area, and 
relationship of funded course of 
study to existing courses of 
study at the submitting IHE. 

3. How many scholars enrolled in 
the funded courses of study, 
completed their programs, or 
dropped out before completion? 

IHE Course of Study Survey 

Counts, percentages, and means 
of individuals enrolled in 
courses of study and completing 
the course or dropping out, 
disaggregated by PDP 
priority/focus area and PDP 
award fiscal year. 

4. What were the quality and 
relevance/usefulness of new or 
significantly modified 
components for funded courses 
of study? 

IHE Course of Study Survey to 
identify components; rubric- 
based reviews, by expert panel, 
of materials documenting new 
and significantly modified 
components within funded 
courses of study. 

Counts and means of 
components, disaggregated by 
PDP priority/focus areas. 
Expert panelist ratings of the 
quality and relevance/ 
usefulness of sampled 
components, along with 
estimates of inter-rater 
agreement. Means and 
distribution of ratings, 
disaggregated by PDP 
priority/focus area and 
component type. 

5. What became of courses of 
study that did not receive PDP 
training grant funding? 

Review of nonfiinded 
applications, IHE Course of 
Study Survey. 

Counts, percentages, and 
means, disaggregated by PDP 
priority/focus area, PDP 
application fiscal year, and 
course of study maintenance/ 
development. 
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Sampling of Training Grant Applications 

The substudy of PDP training grant applicants was conducted as a census of applicants to 
the FY 2006 and FY 2007 PDP grant competitions in the Leadership, Combined, and (FY 2007 
only) Preservice Training Improvement priority areas. Thus, the initial study sample consisted of 
all 537 applications: 190 that were funded by the PDP and 347 that were not funded. Six 
nonfunded applications were removed from the study sample because the evaluation study team 
was unable to identify either the project director or the course of study within the application due 
to missing information. The remaining 190 funded and 341 nonfunded applications formed the 
final study sample. 

Reviews of applications before survey recruitment showed that 53 FY 2007 applications 
(2 1 funded and 32 nonfunded) in the sample were requests for funding for a course of study that 
was also at the center of an FY 2006 nonfunded application. Because the focus of the survey was 
the course of study, not the application, the matched prior nonfunded applications were removed 
from the sample, leaving 288 nonfunded applications in the survey sample. In July 2009, two 
funded applications and two nonfunded applications were used in a final pilot of survey 
questions; none of these respondents was included in the final survey. The sample at the 
initiation of survey recruitment thus consisted of 188 funded applications and 286 nonfunded 
applications. 

During the recruitment process, we found 37 additional FY 2007 applications to be 
requests for funding for a course of study that was also the focus of an FY 2006 nonfunded 
application in the sample. We removed matched, prior applications from the survey pool and 
asked project directors (or their replacements) to submit a survey regarding only the subsequent 
application. Thus, the final survey sample consisted of 437 applications — 188 funded and 249 
nonfunded. 

Review of Extant Data 

The evaluation study team used data from several publicly available and specially 
obtained resources throughout this study. During the design of the PDP evaluation, we 
downloaded abstracts from ED’s Grant Award Database related to 173 training grants from 
FY 2005 and FY 2006 Leadership and Combined priority area competitions. Information about 
the clustering of grant awards within institutions was collected as design decisions were made. 

At the initiation of the evaluation, we obtained copies of funded and nonfunded applications to 
the FY 2006 and FY 2007 Leadership, Combined, and (FY 2007 only) Preservice Training 
Improvement priority areas from OSEP. We carefully reviewed applications to inform 
instrument development and used details from the applications to establish a comprehensive 
picture of proposed courses of study within funded and nonfunded applications. This information 
became the basis for customizing each respondent’s online survey and assisted evaluation study 
team members during respondent recruitment. In addition, we categorized and coded information 
on proposed courses of study during data analysis. 

The evaluation study team also gathered extant data from IES’s Integrated Post- 
secondary Education Data System (IPEDS) and the Monarch Center Discretionary Awards list. 
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We used IPEDS to download infonnation on the population of training programs in the United 
States. We identified programs from academic years 2005 through 2009 that provided 
certificates/degrees and focus areas aligned with the PDP Leadership, Combined, and Preservice 
Improvement priority areas (see chapter 1). IPEDS data were also downloaded to identify IHEs 
in the study sample as doctorate-granting or non-doctorate-granting and as public or private, 
while data from the Monarch Center Discretionary Awards list were used to detennine the 
minority status of IHEs in the study (see chapter 6). 9 

We gathered additional extant data from the ED Grant Award Database; the online 
Discretionary Grants Public Database accessed through the OSEP TA&D Network website; 
Federal Register Online via U.S. Government Printing Office Access; and OSEP PDP training 
grants application slates from FY 2005 through FY 2010. Specifically, we collected data by 
priority/focus area on the number of training grant applications reviewed by OSEP each year and 
the percentage of reviewed applications funded each year (see chapter 1). We also analyzed 
FY 2001 through FY 2007 abstracts from ED and TA&D grant databases to identify funded 
courses of study that were submitted by IHEs with PDP funding in the same priority/focus area 
within the prior 4 years. Finally, we reviewed OSEP PDP application slates from FY 2006 
through FY 2010 to determine the percentage of IHEs reapplying for PDP funding in the same 
priority/focus area in subsequent years, and the percentage funded through those reapplications. 
These findings appear in chapter 6. 

Institution of Higher Education Course of Study Survey 

From October 2009 to March 2010, the evaluation study team administered a web-based 
survey to funded and nonfunded applicants to the FY 2006 and FY 2007 Leadership, Combined, 
and (FY 2007 only) Preservice Training Improvement PDP grant competitions. 

Survey Content and Development 

The IHE Course of Study survey (included as appendix D) was designed as a primary 
means to answer the study’s research questions. The evaluation study team based the content of 
the survey on the goals and requirements found in the relevant requests for grant applications 
published by the PDP and a careful review of both funded and nonfunded applications. We 
developed the survey through a series of drafts and pilot tests with the intent of maximizing 
responses to survey items while minimizing respondent burden. 

We piloted an initial paper survey draft in November 2008 with three randomly selected 
courses of study that had been funded in the FY 2005 PDP competitions and thus were not part 
of the study. This pilot resulted in the clarification of certain questions and the removal of others 
to reduce the burden on respondents. In January 2009, a second pilot was conducted with another 


9 Public institutions are those whose programs and activities were operated by publicly elected or appointed school officials and 
which were supported primarily by public funds at the time of PDP application; doctorate-granting institutions are those that 
offered the doctor’s degree at the time of PDP application; and minority institutions are institutions that, as of 2010 (1) had a 25 
percent or more minority enrollment (excluding the other or nonresident categories), (2) had a historical distinction as minority 
institutions, or (3) were designed to serve special populations. 
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nine randomly selected respondents from the FY 2005 competitions, again using a paper version 
of the survey. Both funded and nonfunded applicants were selected, as well as applicants across 
priorities. Pilot participants were not part of the study. The survey was revised and presented in 
the Federal Register in February 2009. At this point, the survey comprised four questionnaires: 
one for Combined and Preservice Training Improvement priority area grantees, one for 
Combined and Preservice Training Improvement priority area nonfunded applicants, one for 
Leadership grantees, and one for Leadership nonfunded applicants. 

A third pilot was conducted in July 2009 to determine the best approach for surveying 
respondents whose applications included multiple courses of study. We selected four 
applications from FY 2006 and FY 2007 that were representative of the types of multiple course 
of study configurations found in the survey pool. Project directors completed various versions of 
the paper survey and participated in debriefing conversations about the different approaches. 
Pilot respondents also provided additional feedback about select survey items, which was used to 
further refine the survey prior to data collection. Pilot participants were not part of the study. 

The final survey was web-based, allowing for the customization of the survey to each 
application. Respondents followed different paths based on grant competition year, grant priority 
area, application funding status, the number of courses of study within the application, and 
responses to initial survey questions. Surveys for applications that incorporated multiple courses 
of study focused mainly on a single predominant course of study. The predominant course of 
study was detennined by prescreening the applications and was then confirmed in conversation 
with the project director. For funded applications, the predominant course of study was defined 
as the course of study that received the largest amount of grant funds in academic year 2008-09. 
For nonfunded applications, the predominant course of study was defined as the course of study 
for which the most funding was requested in the application. A small number of questions in the 
survey addressed any non-predominant courses of study in the application. 

All survey respondents were asked about the relationship of the proposed course of study 
to an existing course of study. Nonfunded respondents were asked to provide information on the 
status of the proposed course of study following the time of award decision. Funded respondents, 
as well as those nonfunded respondents whose courses of study were developed or maintained 
despite the lack of PDP funding, also were asked to provide the following infonnation related to 
the course of study as it existed in the 2008-09 academic year: (1) description; (2) areas of focus; 
(3) entry and completion requirements; (4) monetary support for scholars; (5) changes to courses 
and other components in the course of study since the time of the application; (6) scholar 
enrollment, completion, and dropout information as reported by applicants from their own 
administrative records or other data collection; (7) standardized exit exams candidates were 
required to take in order to demonstrate knowledge and skills for completion of the course of 
study; (8) fonnal data collection from or about program completers; and (9) for funded 
respondents, their allocation of grant funds. We focused on the academic year 2008-09 because 
the funded applicants for both years (FY 2006 and FY 2007) would have had sufficient time to 
establish the program as proposed in their grant application by then and because it was the 
academic year immediately preceding the data collection. The survey collected 2 or 3 years’ 
prior data from respondents related to course of study enrollment, completion, and standardized 
exit exams. Finally, applicants to the Combined and Preservice Training Improvement priority 
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areas were asked about the fieldwork components of their courses of study. Applicants to the 
Leadership priority were not asked about fieldwork because that is not a regular component of 
doctoral programs. 

Collection of Survey Data 

Among the 188 funded and 249 nonfunded applications that constituted the sample for 
the IHE Course of Study survey, the evaluation study team identified 387 unique project 
directors. 10 Project directors were recruited from October 2009 to March 2010 via postal mail, 
telephone, and electronic mail. Participation in this data collection was mandatory for funded 
applicants because grantees under IDEA are required to participate in program evaluation 
activities (20 U.S.C. 1221e-3 and 3474). Participation was voluntary for nonfunded applicants, 
so the evaluation study team developed specific plans to ensure their successful recruitment. We 
applied five recruitment strategies in particular: (1) extensive use of senior study staff 
throughout recruitment, (2) personalization of all contact with respondents (including 
introductory letters sent from the Department to begin recruitment), (3) use of a customized web 
survey with features that shortened response times and encouraged successful completion, (4) 
use of a customized online management system that ensured regular and timely followup with 
non- or partial responders, (5) individualized TA through a toll-free telephone line answered by a 
senior staff member. Additionally, project directors for nonfunded applications were provided 
with a nominal $30 incentive for completing a survey. 

Data collection took place from October 2009 to March 2010. Surveys were submitted 
for 186 (99 percent) of the funded applications and for 230 (92 percent) of the nonfunded 
applications, for an overall response rate of 95 percent. Of the 416 submitted surveys, 397 (95 
percent) were considered complete, meaning at least 80 percent of the items relevant to that 
respondent were fully answered. Many survey items included a large number of sub-items; an 
item was considered fully answered only when all sub-items were completed. The mean item 
completion rate across all submitted surveys was 97 percent. Of the 19 submitted surveys 
considered incomplete, 7 were for funded applications, while 12 were for nonfunded 
applications. Among incomplete surveys, the range of fully answered items was 14 to 79 percent, 
with a mean of 59 percent. 

During the course of data analysis, evaluation study team members modified select 
survey data based on a systematic data cleaning process. Analysts reviewed survey data for 
internal inconsistencies, outliers, and other problems and, in select instances, followed up with 
respondents for data verification. Based on this review process, we made modifications to 143 
surveys in relation to one or more of the following: (a) the relationship of the proposed course of 
study to an existing course of study, (b) whether the course of study was developed or 
maintained despite lack of PDP funding, (c) significant changes made to courses of study since 
the time of application, and (d) enrollment and completion data. Following data cleaning, 
completion rates changed for 30 surveys. For the 416 submitted surveys, 399 (96 percent) were 
now considered complete. The mean item completion rate across all surveys remained at 97 
percent. Of the now 17 submitted surveys considered incomplete, 7 were for funded applications, 


10 Some project directors submitted more than one application. 
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while 10 were for nonfunded applications. Among incomplete surveys, the range of fully 
answered items was 14 to 76 percent, with a mean of 57 percent. 

Analysis of Survey Data 

All data for PDP-funded courses of study were analyzed separately from data for 
nonfunded courses of study. Within these funding sub-groups, we aggregated data across 
surveys, and disaggregated by PDP priori ty/focus area. For certain descriptive analyses, we also 
disaggregated data by PDP competition fiscal year, by PDP funding history, and by proposed 
course of study maintenance/development. In instances in which applications incorporated 
multiple courses of study, the analysis focused on predominant courses of study. 

Two items on the IHE Course of Study survey asked respondents to identify significant 
changes made to their courses of study since the time of their application. Survey responses for 
both funded and nonfunded courses of study for these items were reviewed and coded according 
to the type and status of change reported. During survey design, the evaluation study team had 
reviewed a random sample of 100 applications with respect to the kinds of changes proposed 
within applications. Based on this review, six types of change components were initially 
identified: courses or classes, training units or modules, faculty, reorganized courses of study, 
mentoring programs, and recruitment plans. We added a seventh category of fieldwork because 
of the large number of surveys that reported changes that specified student teaching, practicum, 
or some other form of fieldwork. Reported changes that did not fall in these categories were 
categorized as other. Each reported change was also coded according to a status category: new to 
the IHE, modification to an existing component, addition of a component that already existed 
elsewhere within the IHE, or a discontinued component. 1 1 

During data analysis, the evaluation study team made adjustments to the counts of new 
and significantly modified components reported by 23 respondents based on infonnation gained 
during the development of a sampling frame for the expert review panel. As described under 
Collection of Materials for Sampled Components below, during the collection of materials from 
funded respondents, we detennined some sampled components to be ineligible for use by the 
expert review panel. Because of this detennination, reported changes for both funded and 
nonfunded applicants were reduced for data analysis. First, we reduced counts for new and 
significantly modified components reported by funded applicants that were not sampled by a 
specific weighted ineligibility fraction. The ineligibility fraction was calculated as the number of 
ineligible components of that type found during the materials collection divided by the total 
number of sampled components of that type, with this fraction weighted based on the probability 
of selection of each of the sampled components. The result was a 4 percent reduction in the count 
of new components and 15 percent reduction in the count of significantly modified components. 
We similarly reduced counts for new and significantly modified components reported by 
nonfunded applicants. The number of new components reported by nonfunded respondents was 
reduced by 7 percent, while the number of significantly modified components was reduced by 17 
percent before data analyses. 


1 1 Ail initial 261 components were each coded for type and status by three members of the evaluation study team. Overall inter- 
rater agreement was 84 percent. Remaining components were coded by a single team member. 
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Limitations of IHE Course of Study Survey 

Because the respondents to the IHE Course of Study Survey were the project directors for 
funded projects and the proposed project directors for nonfunded projects, readers are cautioned 
of the potential for response bias in the self-reported data. Additionally, some of the collected 
data were retrospective, and record keeping by the respondents may have been inconsistent. 


Expert Panel Review of Course of Study Components 

For the substudy of PDP training grant applicants, panels of experts reviewed the quality 
and relevance/usefulness of sampled components that were developed or significantly modified 
by grantees during the funding period. 

Sampling of New and Modified Course of Study Components 

For funded courses of study only, a sample of changed components categorized as new to 
the IHE or modifications to existing components was included in a sample frame for a rubric- 
based review of quality and relevance/usefulness. These components constituted the initial 
sampling frame, with some exceptions. Reported components in the other category were not 
included because their content varied too widely to develop a standard rubric. Changed 
recruitment plans also were not included in the review sample frame because we have data 
regarding the numbers of students actually recruited, which is more meaningful than the rating of 
a plan. Changes that involved reorganized courses of study were not included in the review 
sample frame, because the quantity of materials required for a sound judgment of an item this 
large would place an undue burden on respondents and require too much reviewer time. Finally, 
as described under Collection of Materials for Sampled Components below, some other reported 
components were removed from the sample frame because they were determined by the 
evaluation study team to be ineligible for use in the review. 

As mentioned earlier, adjusted responses to the relevant survey items were gathered from 
23 funded respondents. These reported components, once categorized by type and status, 
constituted the supplementary sampling frame. Before the adjusted responses were gathered, the 
initial and supplementary sampling frames were projected to include 406 new or significantly 
modified courses or classes, 99 new or significantly modified fieldwork components, 32 new or 
significantly modified training units or modules, 32 new faculty, and 28 new or significantly 
modified mentoring programs. For the two largest categories (courses or classes and fieldwork), 
a sampling fraction was chosen that was expected to result in approximately 36 selected cases. 
This sampling fraction was 36/406 = 8.9 percent for courses or classes and 36/99 = 36.4 percent 
for fieldwork components. In the smaller categories, all reported changes were to be sampled. 

The goal was to maximize precision while staying within budget for the review process. 

This process resulted in a number of grantees that would have been asked to provide 
documentation related to as many as six relevant components. While recognizing that there 
would be some cost in statistical efficiency, the evaluation study team decided that the sample 
would have to be modified to ensure that no more than two components would be requested per 
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course of study. In each case in which the initial or supplementary samples contained more than 
two relevant components, we randomly removed sampled components without regard to their 
change type so that no more than two sampled components per course of study remained. The 
reduction in initial and supplemental samples required that an additional sample be drawn to 
compensate for lost components from the two largest categories. The additional sampling frame 
included only funded courses of study from which no components had yet been sampled and 
contained only courses or classes and fieldwork because all other changes had already been 
sampled at a 100 percent rate. The evaluation study team established the goal of reviewing 
similar numbers of fieldwork components and components focused on courses or classes, to 
achieve comparable precision of estimates for each group, not accounting for each group’s 
population size. To reach this goal, all (100 percent) of new or significantly modified fieldwork 
components were sampled to obtain 47 fieldwork components for review. In contrast, 7.7 percent 
of new or significantly modified courses or classes were sampled to obtain 49 courses or classes 
for review. 

Collection of Materials for Sampled Components 

From May to August 2010, the evaluation study team collected documentation of 
sampled components from grantees in preparation for the expert review. We requested specific 
documents and other materials for each component type (table 5.2). During the process of 
materials collection, we found 38 of the 189 requested components to be ineligible for review 
because they were (1) detennined to have existed before the time of the application and to have 

simply been added to the course of study, (2) determined to be duplicative of other reported 

12 

changes, (3) detennined by the evaluation study team to be an insignificant modification, 

(4) reported by the grantee to have not yet been created, or (5) detennined to be unrelated to the 
predominant course of study. Of the 151 eligible components for which materials were 
requested, documentation was obtained for 142 components, for a 94 percent response rate. 
Following a review of materials, we detennined that sufficient documentation was provided for 
valid reviews to occur for 134 components (89 percent of the original request). The final 
distribution of sampled components assigned for expert review was 42 courses or classes, 28 
fieldwork components, 17 new faculty members, 19 mentoring plans, and 28 training units or 
modules. The 134 components were collected from 99 different courses of study. 

Review Panels for Course of Study Components 

The evaluation study team recruited 14 experts from academia with relevant content 
expertise to serve on three-person panels to review course of study components. We identified 
panelists through literature searches, recommendations made by members of the evaluation study 
team and IES staff, and recommendations from potential panelists regarding fellow experts. Each 
panelist was identified, recruited, and assigned to review specific components based on his/her 
knowledge of the component type, topic area of the grant, or grant competition area. Potential 
panelists were screened to prevent conflicts of interest. In June 2010, all review panel members 
were trained during a daylong, in-person training. The specific purposes of the training were to 
inform panelists about their roles and responsibilities, provide an overview of the review process, 


12 


The benchmark for significance in relation to modified components was whether 10 percent or more of the content was new. 
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and enhance the reliability of ratings. Panelists discussed the indicators and anchors of the 
scoring rubrics, practiced scoring sample products using the rubrics, debriefed with their fellow 
panelists, and practiced reconciliation with their colleagues when ratings differed. Panelists also 
provided feedback on the content of the rubrics. 

Table 5.2: Documentation requested, by course of study component type 
Course or class 

1. Assessments of candidates in this course, including information about how the assessments are 
scored (e.g., sample assignments with scoring rubrics), and how assessments are linked to course 
objectives; 

2. Curriculum vitae (CV) of the instructor of this course; 

3. Sample course materials (e.g., 2-3 presentations, 2-3 handouts, and 2-3 assignments); 

4. Syllabus; 

5. Number of students in class; and 

6. Description of how the course differs from the version offered before receipt of the grant (only if 

a modification). 

Fieldwork 

1 . Instruments used to assess candidates, including two sample faculty or supervisor evaluations of 
candidates (please remove candidate names); 

2. CV(s) of faculty primarily responsible for leading and supervising fieldwork; 

3. Fieldwork handbook; 

4. Fieldwork plan and/or syllabus of fieldwork course; 

5. Materials used to train supervisors; and 

6. Description of how the fieldwork differs from the version offered before receipt of the grant 

(only if a modification). 

New faculty member 

1 . CV, including courses taught in the past; 

2. Sample candidate assessments developed or selected by new faculty member, including 
information about how the assessments are scored (e.g., sample assignments with scoring 
rubrics); and 

3. Syllabi of all courses to which the new faculty member was assigned (please indicate if the 

faculty member developed these syllabi). 

Mentoring program 

1 . Mentoring plan; 

2. Materials used to train mentors; 

3. Program data (including number of mentors, number of candidates served, schedule and 
structure of mentor-mentee interactions); and 

4. Description of how the mentoring program differs from the version offered before receipt of the 

grant (only if a modification). 

Training unit or module 

1. CV(s) of key instructors or organizers; 

2. Two or three sample materials from the training unit or module (e.g., agenda, presentations, 
handouts); 

3. Description of the training unit or module; and 

4. Description of how the training unit or module differs from the version offered before receipt of 
the grant (only if a modification). 
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Panelists used documents and other materials to review the 134 new or significantly 
modified components. Each component was reviewed by three experts, for a total of 402 
reviews. Reviews were conducted from July to October 2010. All data were obtained from 
reviewers as planned. Reviews took 2 hours, on average, with another hour (on average) for 
resolution of discrepant ratings on a portion of components. Overall, panelists required an 
average of 43 hours to complete the review process. Adding 8 hours per reviewer for training, 
the typical reviewer committed 5 1 hours to the expert review process. 

Rubrics to Rate Course of Study Components 

Panelists used component-specific rubrics (included as appendix E) developed by the 
evaluation study team to rate the quality and relevance/usefulness only of the particular 
components being reviewed. They did not rate the overall quality or relevance/usefulness of the 
courses of study from which the components were drawn. 

We based the content of the rubrics on the expectations stated in the relevant grant 
priorities developed by OSEP. Specific indicators in the rubrics were li nk ed to application 
review criteria in OSEP’s request for applications, as well as additional research conducted by 
the evaluation study team in the content area. The additional research included a review of 
accreditation documents used by the National Council for the Accreditation of Teacher 
Education. Rubric development underwent a series of external reviews and tests. Initial rubrics 
were developed to assess the nature and quality of new classes, new professors, and other 
changes to funded courses of study. Continued development of the rubrics focused on the 
reduction of respondent burden and resulted in the following modifications to the expert review 
process and corresponding rubrics: (1) documentation would be requested for new and 
significantly modified components; (2) materials would not be requested regarding discontinued 
courses; and (3) the focus of the review would be the quality of the resulting component, not the 
quality of the change that occurred. 

In May 2010, one member of the evaluation’s TWG with expertise in instrumentation and 
teacher education reviewed all five draft rubrics. A second member of the TWG with particular 
expertise in mentoring and fieldwork programs also reviewed the two rubrics corresponding to 
those topics. Based on these reviews, we made revisions to the rubrics. Revised rubrics were then 
piloted with two additional consultants, with expertise in personnel preparation programs, who 
were not members of the TWG and who did not subsequently participate in the actual review. 
Using materials gathered from grantees on a sample of reported components, the consultants 
piloted each rubric, submitted their completed rubrics, and provided written feedback with regard 
to the (1) clarity and appropriateness of the indicators and dimensions for evaluating the 
components, (2) adequacy of collected materials to document each component, (3) reviewer 
confidence in the ratings selected, and (4) length of time needed to review materials and 
complete the ratings. Following the integration of feedback based on the pilot, rubrics were 
presented during the June 2010 training of members of the review panels for course of study 
components. During this training, panelists provided feedback on the content of the rubrics, 
recommending any necessary modifications to indicators or anchors. We used this feedback to 
further refine the rubrics and prepare them for use in the evaluation. 
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Final rubrics shared a similar format across change components consisting of 

(1) indicators rated on a 5-point scale, with 1 being the lowest rating and 5 being the highest; 

(2) provision of anchor descriptions for scores of 1, 3, and 5; (3) an indicator-specific comment 
field provided to record rater’s notes; and (4) a general comment field provided to indicate 
uncertainty about a rating or to note strengths or weaknesses not linked to a specific indicator. 
Rubrics contained common elements of quality and relevance/usefulness that were expected to 
be found across components (e.g., the knowledge and skills emphasized in the course or class 
[fieldwork, etc.] have a basis in rigorous research), as well as unique elements corresponding to 
the particular focus of that change category (e.g., the new faculty member has the necessary 
background and experience to teach the courses to which s/he is assigned). The number of 
indicators for quality and for relevance/usefulness varied by component category based upon the 
nature of each category. Rubrics for two component categories, courses or classes and training 
units or modules, contained indicators grouped into the two dimensions of quality and 
relevance/usefulness; the remaining rubrics, fieldwork, new faculty, and mentoring programs, 
contained indicators grouped into only the single dimension of quality. The number of indicators 
per category ranged from six to nine. The number of quality indicators per category ranged from 
4 to 9, and the number of relevance/usefulness indicators ranged from 0 to 3 for a total of 32 
quality indicators and 6 relevance/usefulness indicators. 

Analysis of Expert Review Data 

The primary tools used for the analysis of expert review data were means and percentage 
distributions. Experts rated sampled course of study components on a series of indicators for two 
dimensions: quality and relevance/usefulness. Each sampled component was rated by three 
expert panel members, making a total of 2,604 ratings on quality indicators and 630 ratings on 
relevance/usefulness indicators. Individual expert ratings for each indicator were averaged across 
raters to create quality and relevance/usefulness scores. Quality means and medians were 
obtained for all components. Relevance/usefulness means and medians were obtained for each 
component categorized as a course or class or training unit or module. We averaged quality and 
relevance/usefulness scores across the set of all relevant components and disaggregated by PDP 
priority/focus area and by component type. Quality and relevance/usefulness scores were 
weighted using the inverse of the probability of selection for each sampled component to allow 
generalization to all components reported by grantees. In addition, we calculated the 
distributions of quality and relevance/usefulness ratings as the percentage of indicator-level 
ratings at each scale value. Percentages were obtained for all components combined, and 
disaggregated by PDP priority/focus area and by component type. 

Inter-rater Agreement for the Review Panels for Course of Study Components 

The evaluation study team used Rwg(j) (James, Demaree, and Wolf 1984) as an index for 
estimating the inter-rater agreement of the expert panel review of course of study components 
and for identifying cases for reconciliation. Rwg(j) ranges from 0 to 1 and estimates the level of 
inter-rater agreement between judges who provide ratings for multiple items for each target (i.e., 


13 For a description of procedures used for sampling reported components, see the section titled Sampling of New and Modified 
Course of Study Components. 
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dimension). Indices were calculated based on unifonn distribution assumptions, that is, raters 
who respond randomly to products have an equal chance of choosing each response option on 
the Likert scale. The following standards were used for interpreting Rwg(j) (LeBreton and Senter 
2008): 0.00-0.30 (lack of agreement), 0.31-0.50 (weak agreement), 0.51-0.70 (moderate 
agreement), 0.71-0.90 (strong agreement), 0.91-1.00 (very strong agreement). Additional 
information on the Rwg(j) statistic is provided as appendix F. 

All 134 components were reviewed for quality and 70 for relevance/usefulness as well. 14 
Mean inter-rater agreements based on initial, individualized ratings were 0.75 for quality (strong 
agreement) and 0.57 for relevance/usefulness (moderate agreement), for an average Rwg(j) of 
0.69 (moderate agreement). Twenty-five components (18.7 percent) had an Rwg(j) of less than 
0.30 for quality, relevance/usefulness, or both. Reconciliation meetings were convened via 
conference call for reviewed components with an Rwg(j) below 0.30. 

We conducted 15 reconciliation calls. A member of the evaluation study team facilitated 
each call. Panelists discussed the strengths and weaknesses of the component in question at the 
dimension (i.e., quality or relevance/usefulness) level and provided rationale for their specific 
ratings on those indicators with the lowest agreement. Expert reviewers had the opportunity to 
resolve discrepancies, although there was no requirement that they reach consensus. Panelists 
informed the evaluation study team of any revised ratings or that no changes were being made, 
via e-mail within 3 business days following the reconciliation call. Following the reconciliation 
process, 1 (0.8 percent) component had an Rwg(j) of less than 0.30 for quality, relevance/ 
usefulness, or both. Mean inter-rater agreements were increased to 0.86 for quality (strong 
agreement) and 0.75 for relevance/usefulness (strong agreement), for an average post- 
reconciliation Rwg(j) of 0.82 (strong agreement). All reviewers completed their independent 
reviews and participated in reconciliation meetings as required, so there were no missing data. 

Limitations of Expert Panel Review of Course of Study Components 

Readers are cautioned that the expert panelist ratings of quality and relevance/usefulness 
of new or significantly modified course of study components only reflect sampled work for 
which review was feasible through the examination of documents or other media. These ratings 
do not reflect the quality or relevance/usefulness of components for which documentation was 
not available. In addition, the ratings do not represent the quality or relevance/usefulness of 
courses of study as a whole, only of components added or significantly changed following the 
awarding of funds from the FY 2006 or FY 2007 PDP training grant competitions. 


14 The rubrics developed to evaluate each type of course of study component (appendix E) originally distinguished some 
indicators as measures of quality, and other indicators as measures of relevance/use fulness. However, during testing of the 
rubrics, expert consultants recommended that it would be more appropriate to classify all of the indicators for components 
involving new faculty, fieldwork, and mentoring programs as quality indicators. 


84 



6. Findings from the Substudy of PDP Training Grant Applicants 


OSEP funds training grants under the PDP to enhance and maintain the supply of 
appropriately trained personnel to improve outcomes for children with disabilities and their 
families. The substudy of PDP training grant applicants was designed to address the following 
research questions: 

1 . What were the characteristics of funded courses of study at IHEs awarded PDP training 
grants in FY 2006 or FY 2007? 

2. How did funded courses of study use PDP training grant funding? 

3. How many scholars 15 enrolled in the funded courses of study, completed their programs, or 
dropped out before completion? 

4. What were the quality and relevance/usefulness of new or significantly modified components 
for funded courses of study? 

5. What became of courses of study that did not receive PDP training grant funding? 

Analyses were based on data gathered through (1) review of relevant extant data, (2) a 
survey of applicants to the FY 2006 and FY 2007 PDP grant competitions, and (3) expert panel 
review of sampled components within funded courses of study. Findings are presented below by 
research question. 

What Were the Characteristics of Funded Courses of Study at IHEs Awarded 
PDP Training Grants in FY 2006 or FY 2007? 

Of the 537 applications from IHEs to the FY 2006 and FY 2007 PDP training grant 
competitions, 190 were funded. Where extant data were used, all 190 funded courses of study are 
described. Four of the 190 funded applicants did not complete the IHE Course of Study Survey 
(included in appendix D) and were therefore not included in analyses of survey data. 

For every priority area in the FY 2006 and FY 2007 PDP training grant competitions, a 
majority of funded applications were from public institutions, and a majority of funded 
applications were from doctorate-granting IHEs (table 6.1). Across all priority areas, 84 percent 
of funded applications were from public institutions; 88 percent were from doctorate-granting 
institutions; and 48 percent were from minority institutions. 


15 Scholar means an individual who is pursuing a degree, license, endorsement, or certification related to special education, 
related services, or early intervention services. 
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Table 6.1: Characteristics of institutions of higher education submitting training grant 
applications for FY 2006 and FY 2007 that were funded by the PDP, by 
priority/focus area 



Percent of funded applications from 



Public 

institutions 

Doctorate-granting 

institutions 

Minority 

institutions 

All priority/focus areas (n=190) 

84 

88 

48 

Leadership (n=46) 

87 

98 

48 

Early Childhood (n=25) 

88 

92 

36 

Low Incidence (n=32) 

84 

100 

44 

High Incidence (FY 2006) (n=13) 

92 

92 

46 

Preservice Improvement 
(High Incidence FY 2007) (n=22) 

95 

82 

41 

Related Services (n=26) 

73 

88 

23 

Minority Institutions (n=26) 

73 

58 

100 


TABLE READS: Among all 190 applications funded through FY 2006 and FY 2007 PDP training grant competitions, 84 
percent were submitted by institutions classified as public at the time of application, 88 percent were submitted by 
institutions granting doctor’s degrees at the time of application, and 48 percent were submitted by institutions classified with 
Minority status in 2010. 

NOTE: Funded applications were classified among all three subcategories for IHE institutional characteristics: Public 
institutions are those whose programs and activities were operated by publicly elected or appointed school officials and that 
were supported primarily by public funds at the time of PDP application; doctorate-granting institutions are those that 
offered the doctor’s degree at the time of PDP application; and minority institutions are institutions that, as of 2010 (1) had a 
25 percent or more minority enrollment (excluding the other or nonresident categories), (2) had a historical distinction as 
minority institutions, or (3) were designed to serve special populations. 

SOURCE: IES Integrated Post-secondary Education Data System (IPEDS) database, and the National Outreach and 
Technical Assistance Center on Discretionary Awards for Minority Institutions (Monarch Center). 


Across FY 2006 and FY 2007 PDP competitions, 37 percent of applications from public 
institutions were funded, versus 32 percent of applications from private institutions (figure 6.1). 
Thirty-eight percent of applications from doctorate-granting institutions were funded, versus 
25 percent of applications from non-doctorate-granting IHEs. Thirty-eight percent of 
applications from minority institutions were funded compared to 34 percent of applications from 
non-minority institutions. 
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Figure 6.1: Percent of FY 2006 and FY 2007 applications funded, by institution of higher 
education characteristic 

Percent of applications 

50 
45 
40 
35 
30 
25 
20 
15 
10 
5 
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FIGURE READS: Across the FY 2006 and FY 2007 PDP competitions, 37 percent of applications from public IHEs were 
funded compared to 32 percent of applications from private IHEs; 38 percent of applications from doctorate-granting IHEs 
were funded compared to 25 percent from non-doctorate-granting IHEs; and 38 percent of applications from minority 
institutions were funded compared to 34 percent of applications from non-minority institutions. 

NOTE: The initial study sample included 537 applications. Six nonfunded applications were removed from the sample because 
the evaluation study team was unable to identify either the project director or the course of study within the application, resulting 
in a study sample of 531 applications. Funded applications were classified among all three subcategories for IHE institutional 
characteristics: Public institutions are those whose programs and activities were operated by publicly elected or appointed school 
officials and that were supported primarily by public funds at the time of PDP application; doctorate-granting institutions are 
those that offered the doctor’s degree at the time of PDP application; and minority institutions are defined as institutions that, as 
of 2010, (1) had a 25 percent or more minority enrollment (excluding the other or nonresident categories), (2) had a historical 
distinction as minority institutions, or (3) were designed to serve special populations. 

SOURCE: IES Integrated Post-secondary Education Data System (IPEDS) database, and the National Outreach and Technical 
Assistance Center on Discretionary Awards for Minority Institutions (Monarch Center). 
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Across all priority/focus areas, 65 percent of funded IHEs in FY 2006 or FY 2007 had 
PDP funding in the same priority/focus area during the prior 4 years (table 6.2). Previous funding 
in the same priority/focus area was highest for funded applicants in the area of High Incidence 
(85 percent) and lowest for funded applicants in the area of Early Childhood (40 percent). 
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Table 6.2: Institution of higher education PDP funding history of PDP applicants funded 
in FY 2006 or FY 2007 



Percent of funded applicants with PDP 
funding in the same priority/focus area 
during any of the prior 4 years 

All priority/focus areas (n=190) 

65 

Leadership (n=46) 

76 

Early Childhood (n=25) 

40 

Low Incidence (n=32) 

78 

High Incidence (FY 2006; (n=13) 

85 

Preservice Improvement 
(High Incidence FY 2007) (n=22) 

59 

Related Services (n=26) 

58 

Minority Institutions (n=26) 

58 


TABLE READS: Among all 190 applications funded through FY 2006 and FY 2007 PDP training grant 
competitions, 65 percent were from UTEs with PDP funding in the same priority/focus area during any of the 
prior 4 years. 

SOURCE: U.S. Department of Education Grant Award Database, Office of Special Education Programs Discretionary 
Grants database. 


Of the 190 applications funded in FY 2006 and FY 2007, 1 1 percent contained multiple 
courses of study. Where multiple courses of study were proposed within a single application, we 
identified the predominant course of study, defined as the course of study that received the 
largest amount of grant funds in AY 2008-09. When the term “course of study” is used 
throughout the sections that follow, it is in reference to the predominant course of study within 
an application. 

Seventy-eight percent of the courses of study funded in the FY 2006 or FY 2007 
competitions proposed to utilize an instructional setting on a campus or other site where face-to- 
face instruction would occur (table 6.3). Overall, 45 percent of the funded courses of study 
proposed to award master’s or educational specialist degrees, and 74 percent proposed to award 
state-issued credentials. Forty-three percent of funded courses of study proposed to prepare 
special education teachers. 

Thirty-seven percent of the funded courses of study proposed to prepare personnel to 
serve all categories of disability, while a combined 34 percent proposed to prepare personnel to 
serve high- or low-incidence disability categories, which comprise multiple disabilities. The 
most common specific areas of disability for which personnel would be prepared were students 
with hearing impairments and students with autism (7 percent of courses of study reporting 
each). Forty-eight percent of funded courses of study proposed to prepare personnel to serve 
general school-age children, while 22 percent proposed to prepare personnel to serve birth to 
preschool and another 22 percent to prepare personnel to serve birth to 2 1 or an unspecified age. 
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Table 6.3: Characteristics of proposed predominant courses of study funded by the PDP in FY 2006 or FY 2007, by PDP 
priority/focus area 


All 

priority/ 


Early 

Low 

High 

Incidence 

Preservice 
Improvement 
(High Incidence 

Related 

Minority 

focus areas 

Leadership 

Childhood 

Incidence 

(FY 2006) 

FY 2007) 

Services 

Institutions 

(n=190) 

(n=46) 

(n=25) 

(n=32) 

(n=13) 

(n=22) 

(n=26) 

(n=26) 


Percent of funded predominant 

courses of study proposing to utilize a 

specific instructional setting 




On-site (campus or 
off-site locations for 

78 96 

68 

56 

85 

77 

88 

73 

face-to-face instruction) 

Combination of online 
and on-site 

18 4 

24 

34 

8 

23 

8 

27 

Online 

4 0 

8 

9 

8 

0 

4 

0 

Percent of funded predominant 

courses of study proposing to award specific degrees 





No degree 

16 2 

20 

31 

23 

23 

4 

19 

Associate’s or Bachelor’s 

13 0 

12 

19 

Q 

55 

0 

12 

degree 

<3 

Master’s or Educational 
Specialist degree 

45 2 

68 

50 

69 

23 

81 

62 

Doctor’s degree 

26 96 

0 

0 

0 

0 

15 

8 

Percent of funded predominant 

courses of study proposing to award specific credentials 1 





State issued credentials 

74 11 

96 

94 

100 

100 

85 

96 

Professional 
organization credentials 

15 0 

0 

3 

0 

0 

81 

23 

Grantee issued 
endorsement or course 
completion only 

6 4 

4 

13 

0 

5 

8 

4 

No credential identified 

22 87 

4 

0 

0 

0 

4 

0 



Table 6.3: Characteristics of proposed predominant courses of study funded by the PDP in FY 2006 or FY 2007, by PDP 
priority/focus area (continued) 


All 

priority/ 


Early 

Low 

High 

Incidence 

Preservice 
Improvement 
(High Incidence 

Related 

Minority 

focus areas 

Leadership 

Childhood 

Incidence 

(FY 2006) 

FY 2007) 

Services 

Institutions 

(n=190) 

(n=46) 

(n=25) 

(n=32) 

(n=13) 

(n=22) 

(n=26) 

(n=26) 


Percent of funded predominant 

courses 

of study proposing to prepare personnel for particular roles 




Dually certified 
general/special educators 

2 

0 

0 

0 

0 

0 

0 

12 

Early intervention or earh 
childhood specialist 

12 

0 

84 

3 

0 

0 

0 

4 

Leadership personnel 

24 

100 

0 

0 

0 

0 

0 

0 

Psychologists 

3 

0 

0 

0 

0 

0 

15 

8 

Related Services personm 
(excluding Speech- 
Language) 

7 

0 

0 

13 

0 

0 

35 

4 

Special education teacher 

43 

0 

16 

84 

100 

100 

0 

62 

Speech-language 

pathologists 

8 

0 

0 

0 

0 

0 

50 

12 

Percent of funded predominant 

courses 

of study proposing to prepare personnel to serve 

particular disability 

areas 2 



All disabilities 

37 

57 

80 

0 

0 

0 

31 

62 

High incidence 

22 

7 

0 

0 

77 

100 

8 

15 

Low incidence 

12 

2 

0 

47 

0 

0 

12 

12 

Autism 

7 

7 

8 

19 

0 

0 

4 

4 

Deaf-Blindness 

1 

0 

0 

3 

0 

0 

0 

4 

Developmental delay 

1 

0 

4 

0 

0 

0 

0 

0 

Emotional Disturbance 

5 

11 

0 

0 

23 

0 

0 

4 



Table 6.3: Characteristics of proposed predominant courses of study funded by the PDP in FY 2006 or FY 2007, by PDP 
priority/focus area (continued) 



All 
priority/ 
focus areas 
(n=190) 

Leadership 

(n=46) 

Early 

Childhood 

(n=25) 

Low 

Incidence 

(n=32) 

High 
Incidence 
(FY 2006) 
(n=13) 

Preservice 
Improvement 
(High Incidence 
FY 2007) 
(n=22) 

Related 

Services 

(n=26) 

Minority 

Institutions 

(n=26) 

Hearing impairment 

7 

0 

16 

16 

0 

0 

15 

4 

Intellectual Disability 

2 

2 

0 

3 

0 

0 

0 

4 

Other health impairment, 
including attention deficit 
disorder 

1 

2 

0 

0 

0 

0 

0 

0 

Specific learning 
disabilities 

5 

9 

0 

0 

8 

0 

15 

0 

Speech-Language 

impairment 

6 

7 

0 

0 

0 

0 

27 

8 

Traumatic brain injury 

1 

0 

0 

3 

0 

0 

4 

0 

Visual impairment 

3 

0 

4 

16 

0 

0 

0 

0 

Percent of funded predominant courses 
Birth to pre-school 22 

of study proposing to prepare personnel to serve particular grade/age levels 3 
24 100 3 0 0 

12 

8 

Elementary school 

7 

2 

12 

0 

23 

5 

0 

19 

Secondary 

4 

2 

0 

3 

8 

5 

0 

15 

Transition or post- 
secondary 

3 

0 

0 

16 

0 

0 

4 

0 

Birth to 21 (or unspecifiec 
age) 

22 

22 

0 

19 

8 

0 

62 

31 

General school age 

48 

52 

0 

69 

62 

95 

23 

38 



Table 6.3: Characteristics of proposed predominant courses of study funded by the PDP in FY 2006 or FY 2007, by PDP 
priority/focus area (continued) 


TABLE READS: Among all 190 applications funded through the FY 2006 or FY 2007 PDP training grant competitions, 78 percent of predominant courses of study proposed on- 
site instruction; 18 percent proposed a combination of on-site and online instruction; and 4 percent proposed online instruction. No degree was offered by 16 percent of these 
predominant courses of study; 13 percent offered an associate’s or bachelor’s degree; 45 percent offered a master’s or educational specialist degree, and 26 percent offered a 
doctor’s degree.; 

'Characteristics of funded predominant courses of study were classified by up to three categories for credential to be awarded 
Characteristics of funded predominant courses of study were classified by up to two categories for disability area to be served. 

Characteristics of funded predominant courses of study were classified by up to two categories for age/grade level to be served. 

NOTE: Surveys of applications that incorporated multiple courses of study focused on a single, predominant course of study, with a small number of questions addressing the other 
courses of study that were included in the application. Data presented at the course of study level represent the predominant course of study. 

SOURCE: IE1E Course of Study Survey - item 5 (appendix D), administered in fall 2009 through winter 2010; review of fiscal year 2006 and 2007 applications, IES Integrated 
Post-secondary Education Data System (IPEDS) database, and the National Outreach and Technical Assistance Center on Discretionary Awards for Minority Institutions (Monarch 
Center). 
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Across all priority/focus areas, 53 percent of funded courses of study differed 
substantially from any course of study at the submitting IHE at the time of PDP application 
(table 6.4). The remaining 47 percent of the funded courses of study were existing courses of 
study that were supported or expanded through the PDP grant. The likelihood of a funded course 
of study differing substantially from any existing course of study at the submitting IHE was 
highest in the area of Early Childhood (67 percent), and lowest in the area of Preservice 
Improvement (36 percent). 


Table 6.4: Percent of predominant courses of study funded in FY 2006 or FY 2007 that 
were substantially different from any existing course of study, by PDP 
priority/focus area 



Percent of funded predominant courses 
of study that were substantially different 
from any existing course of study 

All priority/focus areas (n=186) 

53 

Leadership (n=43) 

51 

Early Childhood (n= 24) 

67 

Low Incidence (n= 32) 

50 

High Incidence (FY 2006) (n=13) 

46 

Preservice Improvement 
(High Incidence FY 2007) (n=22) 

36 

Related Services (n=26) 

58 

Minority Institutions (n=26) 

58 


TABLE READS: Among the 186 predominant courses of study funded in FY 2006 or FY 2007, 53 percent were 
substantially different from any existing course of study at the submitting IHE at the time of PDP application. 

NOTE: Substantially different courses of study were those identified by project directors as differing from any courses of 
study that existed at the institution of higher education! IHE) on at least one of the following features: role or roles for which 
candidates are prepared, degrees or credentials candidates seek, or instructional setting through which course of study is 
offered. Four of the 190 funded applicants from FY 2006 and FY 2007 competitions did not complete the IHE Course of 
Study Survey, thus are not included in any analyses utilizing survey data. 

SOURCE: IHE Course of Study Survey - Item 1 (appendix D), administered in fall 2009 through winter 2010. 


How did funded courses of study use PDP training grant funding? 

During AY 2008-09, grantees varied in use of PDP funds, admission requirements, 
course of study focus, course of study modifications, fieldwork requirements, completion 
requirements, scholar enrollment and dropout, scholar completion, and completer follow-up 
practices. 

Grantee use of PDP funds and monetary support for scholars 

All priorities/focus areas, with the exception of the Preservice Improvement (High 
Incidence FY 2007), required monetary support for scholars. This support could include waived 
tuition and fees as well as the provision of funds for living, travel, and other expenses. Across all 
priority/focus areas where scholar stipends were required, an overall average of 73 percent of 
grant dollars went to monetary support for scholars during academic year 2008-09, and 18 
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percent supported faculty (table 6.5). Among Preservice Improvement grant recipients, which 
were prohibited from spending PDP grant funds for scholar stipends, an average of 61 percent of 
grant dollars were spent on faculty during AY 2008-09. 


Table 6.5: Funding allocations per predominant course of study funded in FY 2006 or FY 
2007, by PDP priority/focus area: Academic year 2008-09 


Mean total amount of 
grant dollars spent during 
academic year (AY) 


Mean percent of grant dollars spent on 



2008-09 

Curriculum 

Faculty 

Scholars 1 

Other 

All priority/focus areas 
requiring stipends (n=159) 

$157,016 

2 

18 

73 

7 

Leadership (n=43) 

$159,974 

1 

13 

80 

5 

Early Childhood (n=24) 

$157,936 

2 

20 

72 

7 

Low Incidence (n=31) 

$158,120 

3 

24 

64 

9 

High Incidence (FY 2006) 
(n=12) 

$178,818 

1 

21 

75 

4 

Related Services (n=25) 

$150,208 

2 

17 

73 

9 

Minority Institutions (n=24) 

$145,558 

1 

18 

72 

8 

Preservice Improvement 
(High Incidence FY 2007) 

$77,197 

14 

61 

0 

25 


(n=21) 


TABLE READS: Among 159 predominant courses of study funded in FY 2006 or FY 2007 where scholar stipends were 
required, the mean total amount of grant funds spent during AY 2008-09 was $157,016 . Of these funds, a mean of 2 
percent was spent on curriculum, 18 percent on faculty, 73 percent on scholars, and 7 percent on other expenses. 

’The Preservice Improvement (High Incidence FY 2007) priority/focus area did not permit grant funds to be spent on 
scholar stipends. All other areas required that some grant funds be spent on scholar stipends. 

NOTE: One funded course of study in the Preservice Improvement (High Incidence FY 2007) priority area had not yet been 
developed by AY 2008-09. Five funded course of study did not complete survey item 22. Data from these courses of study 
are therefore not included in analyses for this table. Detail may not sum to totals due to rounding. 

SOURCE: IHE Course of Study Survey - Item 22 (appendix D), administered in fall 2009 through winter 2010. 


On average, across all priority/focus areas where scholar stipends were required, 

88 percent of scholars enrolled in a grant-supported course of study received monetary support 
from the PDP grant in AY 2008-09, and the mean amount of that support was $1 1,558 (table 
6.6). Leadership grantees supported the fewest scholars on average and provided the largest 
mean amount of monetary support. An average of 20 percent of scholars enrolled in funded 
courses of study received monetary support from other sources, and that support averaged 
$8,701. 
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Table 6.6: Monetary support for scholars per predominant course of study funded in FY 

2006 or FY 2007, by PDP priority/focus area: Academic year 2008-09 



Mean 
number of 
scholars 
enrolled 

Mean 
percent of 
scholars 
receiving 
grant 
support 

Mean 
amount of 
grant 
support 

Mean 
percent of 
scholars 
receiving 
other 
monetary 
support 

Mean 
amount 
of other 
support 

All priority/ focus areas with 
required stipends (n=161) 

18.2 

88 

$11,558 

20 

$8,701 

Leadership (n=43) 

9.1 

87 

$21,366 

31 

$12,117 

Early Childhood (n=24) 

20.0 

86 

$7,480 

13 

$1,627 

Low Incidence (n=31) 

22.0 

89 

$6,465 

15 

$5,424 

High Incidence (FY 2006) 
(n=12) 

23.1 

94 

$7,246 

9 

$2,453 

Related Services (n=25) 

19.7 

83 

$11,931 

17 

$10,336 

Minority Institutions (n=26) 

23.2 

90 

$6,628 

19 

$5,962 


TABLE READS: Among 161 predominant courses of study funded in FY 2006 or FY 2007 where scholar stipends were 
required, the mean number of scholars enrolled in 2008-09 was 18.2. The mean percentage of scholars receiving grant 
support was 88 percent; the mean amount of candidate grant support was $1 1,558. The mean percentage of candidates 
receiving other monetary support was 20 percent; the mean amount of candidate support from other funds was $8,70 1 . 

NOTE: Three funded courses of study did not provide relevant data for survey items 15, 16, and 20. Data from these courses 
of study are therefore not included in analyses for this table. Funds from Preservice Improvement grants (High Incidence 
FY 2007) could not be used to cover student stipends, so these courses of study were excluded from this table. Calculations 
for this table are based only on data from the predominant course of study within each grant. 

SOURCE: IHE Course of Study Survey - Items 15, 16, and 20 (appendix D), administered in fall 2009 through winter 2010. 


Admission requirements of funded courses of study 

Among the various criteria used for admitting scholars into funded courses of study, 
minimum GPA was the most common admission requirement (92 percent, table 6.7), followed 
by candidate statement of professional goals (84 percent) and letters of reference (83 percent). 
Among Leadership courses of study, minimum scores on standardized tests (e.g., ACT, SAT, 
GRE) and candidate experience related to the program were also commonly required (88 percent 
and 86 percent, respectively); while among Minority Institution courses of study, candidate 
interviews were also commonly required (92 percent). 
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Table 6.7: Percent of predominant courses of study funded in FY 2006 or FY 2007 that used particular admission criteria, by 

PDP priority/focus area: Academic year 2008-09 



All 
priority/ 
focus areas 
(n=185) 

Leadership 

(n=43) 

Early 

Childhood 

(n=24) 

Low 

Incidence 

(n=32) 

High 
Incidence 
(FY 2006) 
(n=13) 

Preservice 
Improvement 
(High Incidence 
FY 2007) 
(n=21) 

Related 

Services 

(n=26) 

Minority 

Institutions 

(n=26) 

Minimum GPA 

92 

86 

96 

84 

100 

100 

88 

100 

( Mean minimum 
GPA required) 

2.96 

3.26 

2.86 

2.88 

2.94 

2.76 

3.06 

2.88 

Candidate statement of 
professional goals 

84 

93 

79 

84 

77 

48 

96 

96 

Letters of reference or 
recommendation for 
candidate 

83 

93 

79 

84 

92 

52 

88 

85 

Candidate experience 
related to program 

66 

86 

67 

66 

69 

38 

69 

54 

Candidate interview 

61 

79 

33 

50 

62 

33 

58 

92 

Candidate writing 
sample 

57 

70 

54 

59 

54 

38 

46 

62 

Minimum scores on 
standardized test (e.g., 
ACT, SAT, GRE) 

56 

88 

50 

38 

54 

43 

65 

31 

Prerequisite courses or 
fieldwork 

48 

37 

42 

41 

31 

67 

50 

69 

Preadmission portfolio 

38 

65 

21 

25 

46 

24 

46 

27 

Minimum scores on 
PRAXIS 1 Reading, 
Writing, or Math 

13 

0 

21 

19 

23 

43 

0 

4 



Table 6.7: Percent of predominant courses of study funded in FY 2006 or FY 2007 that used particular admission criteria, by 

PDP priority/focus area: Academic year 2008-09 (continued) 


TABLE READS: Among 185 predominant courses of study funded in FY 2006 or FY 2007, 92 percent required a minimum grade point average for admission in academic year 
2008-09 with a mean requirement of 2.96; 84 percent required a statement of professional goals; 83 percent required letters of reference or recommendation for the candidate; 66 
percent required candidate experience related to the program; 61 percent required a candidate interview; 57 percent required a writing sample; 56 percent required minimum scores 
on standardized tests; 48 percent required prerequisite courses or fieldwork; 38 percent required a preadmission portfolio; and 13 percent required minimum scores on PRAXIS 1 
Reading, Writing or Math tests. 

NOTE: One funded course of study in the Preservice Improvement (High Incidence FY 2007) priority area had not yet been developed by academic year 2008-09. Data from this 
course of study are therefore not included in analyses for this table. 

SOURCE: IHE Course of Study Survey - Items 7 and 14 (appendix D), administered in fall 2009 through winter 2010. 



Areas of specified focus among funded courses of study 

All courses of study in the Leadership priority area reported focusing to a moderate or 
great extent on the use of evidence-based practices during AY 2008-09 (table 6.8). Ninety-five 
percent reported such a focus on research methodology, and 93 percent reported such a focus on 
the specialized needs of children from diverse cultural and linguistic backgrounds. 

Among courses of study in the Combined priority area (including the FY 2007 High- 
Incidence focus area), 99 percent reported focusing to a moderate or great extent on the use of 
evidence-based practices (table 6.9). Ninety-five percent reported such a focus on extended field 
experiences; 95 percent reported such a focus on meeting certification, license, or endorsement 
requirements; and 92 percent reported such a focus on the specialized needs of children from 
diverse cultural and linguistic backgrounds. 

Modification of course of study components since PDP funding 

The most commonly reported changes made to funded courses of study following PDP 
funding were new or significantly modified courses or classes (table 6.10). Across all 
priority/focus areas, project directors reported an average of one new course or class and 1.5 
significantly modified courses or classes per funded course of study. Early Childhood grantees 
reported the highest mean number of new courses or classes ( 1 .6 per course of study), while 
Preservice Improvement grantees reported the highest mean number of significantly modified 
courses or classes (4.7 per course of study). Project directors also reported an average of one new 
training unit or module for every five courses of study (0.2 per course of study), and an average 
of one significantly modified fieldwork component for every three courses of study (0.3 per 
course of study). 
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Table 6.8: Percent of predominant courses of study funded in the Leadership area in FY 2006 or FY 2007 that focused to a 
moderate or great extent on specified areas: Academic year 2008-09 

Percent focusing to a moderate or great 


Area of focus — all predominant courses of study funded in the Leadership area (n=43) extent on the area 

Scholar knowledge and skill development 

Use of evidence-based practices in service delivery 100 

Research methodology 95 

Specialized needs of children with disabilities from diverse cultural and linguistic backgrounds 93 

Personnel preparation 84 

Integrated general and special education practice 77 

Policy or advocacy 63 

Special education administration or supervision 28 

Course of study instructional or curricular approaches 

Cultural and linguistic diversity in recruiting scholars 8 1 

Support systems for scholars to enhance retention and success in the program 63 

Field-based training in high-need communities 53 

Meeting certification, license, or endorsement requirements 23 


TABLE READS: Among the 43 predominant courses of study funded in the Leadership priority area in FY 2006 or FY 2007, 100 percent focused to a moderate or great 
extent on use of evidence-based practices in service delivery in academic year 2008-09. 

SOURCE: IHE Course of Study Survey - Item 17 (appendix D), administered in fall 2009 through winter 2010. 



Table 6.9: Percent of predominant courses of study funded in the Combined area in FY 2006 or FY 2007 (including the FY 

2007 High-Incidence focus area) that focused to a moderate or great extent on specified areas, by PDP 
priority/focus area: Academic year 2008-09 

Percent focusing to a moderate or great extent on the area 

Preservice 



All 

Combined 

Early 

Low 

High 

Incidence 

Improvement 
(High Incidence 

Related 

Minority 


priority area 

Childhood 

Incidence 

(FY 2006) 

FY 2007) 

Services 

Institutions 

Area of focus 

(n=142)‘ 

(n=24) 

(n=32) 

(n=13) 

(n=21) 

(n=26) 

(n=26) 

Scholar knowledge and skill development 







Use of evidence-based 
practices in service delivery 
Specialized needs of children 

99 

100 

97 

100 

100 

96 

100 

with disabilities from diverse 
cultural and linguistic 
backgrounds 

92 

88 

91 

100 

95 

85 

96 

Integrated general and special 
education practice 

83 

88 

78 

85 

95 

69 

88 

Cooperation with SEAs, LEAs, 
or Part C lead agencies 
Highly qualified teacher 

74 

71 

88 

85 

81 

50 

81 

requirements of NCLB and 
IDEA 

74 

75 

72 

92 

95 

42 

81 

Involvement of parents in their 








children’s educational 
planning and service delivery 

67 

79 

72 

77 

62 

65 

50 

Course of study instructional or curricular approaches 






Extended field experiences or 
practices 

Meeting certification, license. 

95 

96 

97 

85 

100 

100 

88 

or endorsement 
requirements 

95 

92 

91 

92 

100 

96 

100 

Collecting data on course of 
study quality 

90 

96 

84 

85 

95 

88 

92 

Field-based training in high- 
need communities 

87 

79 

81 

85 

100 

81 

100 



Table 6.9: Percent of predominant courses of study funded in the Combined area in FY 2006 or FY 2007 (including the FY 
2007 High-Incidence focus area) that focused to a moderate or great extent on specified areas, by PDP 
priority/focus area: Academic year 2008-09 (continued) 


Percent focusing to a moderate or great extent on the area 


Area of focus 

All 

Combined 
priority area 
(n=142)' 

Early 

Childhood 

(n=24) 

Low 

Incidence 

(n=32) 

High 
Incidence 
(FY 2006) 
(n=13) 

Preservice 
Improvement 
(High Incidence 
FY 2007) 
(n=21) 

Related 

Services 

(n=26) 

Minority 

Institutions 

(n=26 

State-identified needs for 

87 

100 

75 

92 

95 

81 

88 

highly qualified personnel 
Cultural and linguistic 

diversity in recruiting 
scholars 

Involvement of individuals 
with disabilities or their 

85 

88 

88 

100 

67 

69 

100 

parents in the IHE’s 
program planning, 
implementation, and 
evaluation 

Support systems for scholars 

70 

75 

81 

69 

57 

77 

54 

to enhance retention and 
success in the program 

70 

67 

63 

85 

71 

54 

88 

Mentoring and induction for 
program graduates 

60 

46 

50 

69 

62 

54 

85 

Paraprofessional preparation 

16 

13 

19 

31 

24 

8 

12 


TABLE READS: Among 142 predominant courses of study funded in the Combined priority area in FY 2006 or FY 2007, 99 percent focused to a moderate or great extent on 
use of evidence-based practices in service delivery in academic year 2008-09. 

'including the FY 2007 Fligh-Incidence focus area. 

NOTE: Preservice Improvement grants have been included as a subcategory of the Combined priority area for this table; while these grants represented a distinct priority area 
in FY 2007, the focus of the priority remained High Incidence disabilities. One funded course of study in the Preservice Improvement (High Incidence FY 2007) priority area 
had not yet been developed by academic year 2008-09. Data from this course of study are therefore not included in analyses for this table. 

SEA = state education agency; LEA = local education agency; NCLB = No Child Left Behind; IDEA = Individuals with Disabilities Education Act; IHE = institutions of 
higher education 

SOURCE: IHE Course of Study Survey - Item 17 (appendix D), administered in fall 2009 through winter 2010. 



Table 6.10: Mean number of new or significantly modified components per predominant course of study funded in the 
FY 2006 or FY 2007 PDP training grant competitions, by priority/focus area 


Type of component 

All 

focus/ 

priority 

areas 

(n=182) 

Leadership 

( n =43) 

Early 

Childhood 

( n =24) 

Low 

Incidence 

( n =32) 

High 
Incidence 
(FY 2006) 
(n=12) 

Preservice 
Improvement 
(High Incidence 
FY 2007) 
(n=21) 

Related 

Services 

( n =24) 

Minority 

Institutions 

( n =26) 

All new components 

1.55 

1.43 

2.13 

1.73 

1.21 

1.37 

1.1 

1.68 

Courses or classes 

0.95 

0.85 

1.59 

1.08 

0.82 

0.76 

0.70 

0.80 

Faculty positions 

0.13 

0.11 

0.00 

0.24 

0.08 

0.14 

0.08 

0.22 

Fieldwork 

0.18 

0.19 

0.17 

0.25 

0.22 

0.19 

0.15 

0.05 

Mentoring programs 

0.09 

0.00 

0.20 

0.03 

0.00 

0.18 

0.04 

0.23 

Training units, modules 

0.20 

0.28 

0.17 

0.13 

0.09 

0.10 

0.13 

0.38 

All significantly modified 
components 

1.86 

0.75 

1.96 

1.86 

0.76 

5.47 

1.61 

1.41 

Courses or classes 

1.47 

0.61 

1.49 

1.40 

0.55 

4.74 

1.04 

1.15 

Faculty positions 

0.01 

0.00 

0.00 

0.03 

0.00 

0.00 

0.00 

0.00 

Fieldwork 

0.34 

0.12 

0.39 

0.43 

0.21 

0.68 

0.53 

0.18 

Mentoring programs 

0.03 

0.02 

0.08 

0.00 

0.00 

0.05 

0.04 

0.04 

Training units, modules 

0.01 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.04 


TABLE READS: Among 182 predominant courses of study funded in FY 2006 or FY 2007, an average of 1.55 new components per course of study were reported, including 
averages of 0.95 new courses or classes, 0.13 new faculty positions, 0.18 new fieldwork components, 0.09 new mentoring programs, and 0.20 new training units or modules. 

NOTES: Changes to course of study components reported by PDP grantees were associated with the grant time period, but the data collection did not attempt to establish a 
more substantive connection between reported changes and PDP funding; causal attribution of the changes to grant funds was problematic because new components were 
commonly developed with funding from multiple sources. Because insufficient information was available for coding, some of the components that were coded into the 
sampling frame and then selected for review were, upon review, determined to be ineligible. A few others were re-coded, but still eligible for expert review. This table reflects 
estimates of true eligibility and corrected coding. One funded course of study in the Preservice Improvement (High Incidence FY 2007) priority area had not yet been 
developed by academic year 2008-09. Three funded courses of study did not complete survey items 18 and 19. Data from these courses of study are therefore not included in 
analyses for this table. Individual component means do not sum to all components means due to rounding. 

SOURCE: IHE Course of Study Survey - Items 18 and 19 (appendix D), administered in fall 2009 through winter 2010. 



Fieldwork requirements among funded courses of study 

Ninety-seven percent of courses of study in the Combined priority area (including the FY 
2007 High-Incidence focus area) included a fieldwork component during AY 2008-09 
(table 6.1 1). All courses of study in the Preservice Improvement, Related Services, and Minority 
Institutions areas reported fieldwork components. (Applicants to the Leadership priority were not 
asked about fieldwork because that is not a regular component of doctoral programs.) On 
average, 588 total fieldwork hours were required among those courses of study, and 70 percent 
of fieldwork activities were supervised by faculty. 


Table 6.11: Fieldwork among predominant courses of study funded in the Combined 

priority area in FY 2006 or FY 2007 (including the FY 2007 High-Incidence 
focus area), by PDP focus area: Academic year 2008-09 



Percent of 
courses of 

Among courses of study with fieldwork 
components, mean 

Priority area 

study with 
fieldwork 
components 

Total fieldwork 
hours required 

Percent of fieldwork 
activities supervised by 
faculty 

All Combined (n=142)' 

97 

588 


70 

Early Childhood (n=24) 

96 

425 


61 

Low Incidence (n=32) 

97 

485 


73 

High Incidence (FY 2006) 
(n=13) 

85 

758 


73 

Preservice Improvement 
(High Incidence FY 2007) 
(n=21) 

100 

752 


70 

Related Services (n=26) 

100 

626 


60 

Minority Institutions (n=26) 

100 

612 


81 


TABLE READS: Among 142 predominant courses of study funded in the Combined priority area in FY 2006 or FY 2007, 97 
percent included fieldwork during academic year 2008-09. Among courses of study with fieldwork, the mean total fieldwork 
hours required was 588, and the mean percent of fieldwork activities supervised by faculty was 70 percent. 

1 Including the FY 2007 Fligh-Incidence focus area. 

NOTE: For this analysis, Preservice Improvement grants have been included as a subcategory of the Combined priority area. One 
funded course of study in the Preservice Improvement (High Incidence FY 2007) priority area had not yet been developed by 
academic year 2008-09. Data from this course of study are therefore not included in analyses for this table. Applicants to the 
Leadership priority area were not asked to complete IHE Course of Study Survey item 8. 

SOURCE: IHE Course of Study Survey - Item 8 (appendix D), administered in fall 2009 through winter 2010. 


Completion requirements among funded courses of study 

During AY 2008-09, across all priority/focus areas, the reported mean GPA required for 
course of study completion was 2.96 (table 6.12). The average credit hours required for 
completion ranged from 36 hours for courses of study that did not award a degree to 97 hours for 
courses of study that awarded an associate’s or bachelor’s degree. 
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Table 6.12: Minimum GPA and credit hour requirements for completion of predominant 
courses of study funded in FY 2006 or FY 2007, by degree awarded and PDP 
priority/focus area: Academic year 2008-09 



Mean 

minimum 

Mean credit hours required for completion 
of entire program 



GPA “ 
required for 
completion 

No Degree 1 

Associate’s/ 

Bachelor’s 

Master’s/ 

Specialist Doctor’s 

All priority/focus areas 
(n=185) 

2.96 

36.21 

97.16 

49.89 

78.74 

Leadership (n=43) 

3.04 

t 

t 

36.00 

80.95 

Early Childhood 
(n=24) 

2.90 

28.50 

85.00 

39.76 

t 

I ,ow Incidence ___ 

(n=32) 

2.92 

31.80 

93.00 

47.81 

T 

1 lioh Incidence _ . _ 

(FY 2006) (n=13) 

2.98 

32.67 

90.00 

49.67 

t 

Preservice 

Improvement 
(High Incidence FY 2007) 

(n=21) 

2.86 

43.40 

106.33 

67.25 

t 

Related Services 
(n=26) 

2.88 

18.00 

t 

59.16 

49.00 

Minority Institutions 
(n=26) 

3.01 

49.80 

83.33 

48.38 

78.00 


TABLE READS: Among 185 predominant courses of study funded in FY 2006 or FY 2007, the mean minimum GPA 
required for completion during academic year 2008-09 was 2.96. The mean number of credit hours required for the 
completion of courses of study that offered no degree was 36.21 . For predominant courses of study offering associate’s or 
bachelor’s degrees, the mean was 97.16 credit hours; for courses of study offering master’s or specialist degrees, the mean 
was 49.89 hours, and for courses of study offering doctor’s degrees, the mean was 78.74 credit hours. 

| Not applicable. No courses of study were identified for the category under this priority/focus area. 

1 Courses of study not awarding degrees were typically credential-only programs. 

NOTE: For this table, the Associate’s/Bachelor’s subgroup consisted of seven courses of study awarding associates and 47 
awarding bachelors, while the Master’s/Specialist subgroup consisted of four courses of study awarding Specialist degrees 
and 173 awarding masters. One funded course of study in the Preservice Improvement (High Incidence FY 2007) priority 
area had not yet been developed by academic year 2008-09. Data from this course of study are therefore not included in 
analyses for this table. 

SOURCE: I HE Course of Study Survey - Items 12 and 14 (appendix D), administered in fall 2009 through 
winter 2010. 


Completer follow-up 

Across all priority/focus areas, project directors for 80 percent of funded courses of study 
reported collecting some form of information from or about their completers (table 6.13). A 
survey of completers was the most common means for funded courses of study to collect 
information (74 percent), followed to a lesser extent by a survey of employer or supervisor (37 
percent) and structured interviews with completers (30 percent). Seventy-six percent of courses 
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of study across priority/focus areas collected information on the usefulness of or scholar 
satisfaction with the training program, while 72 percent gathered information on the employment 
status of their completers, including position and location. In contrast, 25 percent collected 
information on employer satisfaction with scholars’ preparation, and 9 percent collected outcome 
data on the students served by completers. 
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Table 6.13: Percent of predominant courses of study funded in FY 2006 or FY 2007 that collected some form of follow-up data 
on or about completers, by PDP priority/focus area 



All 





Preservice 





Early 


High 

Improvement 

Related 

Minority 


priority/ 


Low 

Incidence 

(High Incidence 


focus areas 

Leadership 

Childhood 

Incidence 

(FY 2006) 

FY 2007) 

Services 

Institutions 


(n=184) 

(n=43) 

(n=24) 

(n=31) 

(n=13) 

(n=21) 

(n=26) 

(n=26) 

Percent collecting any 
follow-up data 

80 

70 

83 

71 

77 

100 

77 

92 

Percent using particular methods 








Survey of completers 

74 

65 

83 

68 

77 

90 

69 

81 

Survey of employer 

37 

23 

38 

29 

31 

67 

46 

38 

or supervisor 

Structured interviews 
with completers 

30 

26 

33 

32 

15 

38 

35 

31 

School district or 

9 


A 

3 

15 

29 

A 

12 

state reports 

D 


H 

Structured interviews 
with employer or 
supervisor 

7 

2 

8 

3 

8 

5 

8 

15 

Percent collecting particular information 








Usefulness of, or 
scholar satisfaction 
with, training 
program 

76 

63 

79 

68 

77 

100 

73 

88 

Employment status, 
position, primary 
area, or location 

72 

60 

71 

65 

77 

90 

65 

88 



Table 6.13: Percent of predominant courses of study funded in FY 2006 or FY 2007 that collected some form of follow-up data on or 
about completers, by PDP priority/focus area (continued) 



All 
priority/ 
focus areas 
(n=184) 

Leadership 

(n=43) 

Early 

Childhood 

(n=24) 

Low 

Incidence 

(n=31) 

High 
Incidence 
(FY 2006) 
(n=13) 

Preservice 
Improvement 
(High Incidence 
FY 2007) 
(n=21) 

Related 

Services 

(n=26) 

Minority 

Institutions 

(n=26) 

Ages or grades of 
children served 

47 

16 

67 

52 

62 

67 

38 

58 

Job performance, 

professional honors, 
publications or 
presentations 

Employer satisfaction 

34 

58 

13 

23 

38 

48 

35 

12 

with scholars’ 
preparation 

Outcome data on 

25 

t 

21 

29 

38 

52 

23 

38 

students served by 
completers 

9 

t 

8 

16 

8 

19 

8 

8 


TABLE READS: Among 184 FY 2006 and FY 2007 funded predominant courses of study, 80 percent collect some form of data from or about scholars who have completed 
the course of study, 74 percent conduct surveys of completers, 37 percent survey employers or supervisors, 30 percent use structured interviews with completers, 9 percent 
collect school district or state reports, and 7 percent use structured interviews with employers or supervisors. Additionally, 76 percent collected information about the 
usefulness of or scholar satisfaction with the training program, 72 percent about employment status, 47 percent about the ages or grades of children served, 34 percent about 
job perfonnance and professional honors, 25 percent about employer satisfaction, and 9 percent on outcome data on students served. 

1 Not applicable. Leadership courses of study were not asked to respond to these specific items. 

NOTE: One course of study funded in the Preservice Improvement (High Incidence FY 2007) priority area had not been developed by academic year 2008-09, and one course 
of study funded in the Low Incidence focus area did not complete a survey item (#23). 

SOURCE: IHE Course of Study Survey - Items 23, 24, and 25 (appendix D). 



How many scholars enrolled in the funded courses of study, completed their 
programs, or dropped out before completion? 

PDP grant awards do not correspond to academic years, and start dates for PDP grants are 
contingent on decisions made by the funded IHEs. Start dates among FY 2006 grants ranged 
from August 2006 to January 2007. Start dates for FY 2007 grants ranged from January 2007 to 
January 2008. Nevertheless, data of means for enrollment, completion, and dropping out can be 
examined across years and presented in contrast to the year prior to grant funding. 

Scholar enrollment and completion among funded courses of study 

Table 6. 14 provides 4 years of enrollment and completion data for scholars in FY 2006 
funded courses of study: the academic year prior to grant funding (2005-06), a transitional year 
(2006-07) in which grant activities began, and two academic years (2007-08 and 2008-09) when 
the funded courses of study fully used their grant funding. Table 6.15 shows the corresponding 
data for scholars in FY 2007 funded courses of study. Those data cover 3 years only: the 
academic year prior to grant funding (2006-07), a transitional year (2007-08) in which grant 
activities began, and one academic year when the funded courses of study were operating with 
their full grant funding (2008-09). The number of scholars was counted as zero for those 
academic years for which the project director reported that the course of study did not exist. 

In AY 2008-09, FY 2006 funded courses of study each enrolled 1,607 scholars, averaging 
just over 20 scholars per course of study (table 6. 14). In that same year, FY 2007 funded courses 
of study enrolled 2,592 scholars, averaging slightly less than 26 scholars per course of study 
(table 6.15). Across all priority/focus areas in the FY 2006 and FY 2007 grant competitions, the 
mean number of enrollees, of new scholars, and of degree and credential earners per funded 
courses of study increased from the academic year before funding to the first academic year of 
full grant funding. 

This increase holds across all enrollment and completion variables for each priority/focus 
area as well, with two exceptions: The mean number of scholars earning a primary credential in 
FY 2006 Feadership courses of study remained at zero from 2005-06 to 2007-08 (table 6.14), 
and the mean number of scholars earning a primary degree in FY 2007 Feadership courses of 
study decreased from 2006-07 to 2008-09 (table 6.15). 
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Table 6.14: Total and mean enrollment and completion among scholars in FY 2006 funded 
predominant courses of study, by PDP priority/focus area: Academic years 
2005-06 through 2008-09 




Academic year 




Grant 

Grant 

Grant 


Prior to 

activities 

fully 

fully 


grant 

begin 

utilized 

utilized 

Course of study/type of scholar 

2005-06 

2006-07 

2007-08 

2008-09 

All FY 2006 funded courses of study, all priority/focus areas (n=80) 



Total number of scholars enrolled 

490 

877 

1,334 

1,607 

Mean number of scholars per course of study 

6.13 

10.96 

16.68 

20.09 

Full-time scholars enrolled 

4.94 

8.28 

12.06 

14.20 

Part-time scholars enrolled 

1.19 

2.79 

4.76 

5.89 

Scholars new to the course of study 

2.55 

6.39 

9.89 

9.83 

Scholars who earned the primary degree 

2.48 

3.52 

4.75 

7.22 

Scholars who earned the primary credential 

2.68 

4.25 

6.03 

9.61 

Leadership (n=22) 





Total number of scholars enrolled 

94 

172 

240 

246 

Mean number of scholars per course of study 

4.27 

7.82 

10.91 

11.18 

Full-time scholars enrolled 

4.23 

7.14 

9.55 

9.64 

Part-time scholars enrolled 

0.05 

0.68 

1.36 

1.55 

Scholars new to the course of study 

1.14 

2.77 

4.64 

3.23 

Scholars who earned the primary degree 

0.38 

0.48 

1.19 

1.67 

Scholars who earned the primary credential 

0 

0 

0 

1.00 

Early Childhood (n=10) 





Total number of scholars enrolled 

21 

60 

156 

199 

Mean number of scholars per course of study 

2.10 

6.00 

15.60 

19.90 

Full-time scholars enrolled 

1.00 

3.40 

7.70 

8.30 

Part-time scholars enrolled 

1.10 

2.60 

7.90 

11.60 

Scholars new to the course of study 

0.70 

4.70 

11.40 

11.20 

Scholars who earned the primary degree 

1.33 

3.83 

5.67 

9.33 

Scholars who earned the primary credential 

0 

1.89 

6.78 

10.56 

Low Incidence (n=14) 





Total number of scholars enrolled 

44 

134 

247 

276 

Mean number of scholars per course of study 

3.14 

9.57 

17.64 

19.71 

Full-time scholars enrolled 

2.36 

6.46 

13.62 

13.57 

Part-time scholars enrolled 

0.79 

3.57 

5.00 

6.14 

Scholars new to the course of study 

1.64 

5.86 

10.93 

8.64 

Scholars who earned the primary degree 

1.60 

2.30 

4.30 

6.60 

Scholars who earned the primary credential 

1.57 

2.93 

3.86 

7.86 

High Incidence (FY 2006) (n=12) 





Total number of scholars enrolled 

94 

188 

234 

277 

Mean number of scholars per course of study 

7.83 

15.67 

19.50 

23.08 

Full-time scholars enrolled 

7.67 

12.50 

15.67 

18.50 

Part-time scholars enrolled 

0.17 

3.17 

3.83 

4.58 

Scholars new to the course of study 

4.25 

9.92 

12.25 

11.83 
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Table 6.14: Total and mean enrollment and completion among scholars in FY 2006 funded 
predominant courses of study, by PDP priority/focus area: Academic years 
2005-06 through 2008-09 (continued) 




Academic Year 




Grant 

Grant 

Grant 


Prior to 

activities 

fully 

fully 


grant 

begin 

utilized 

utilized 

Course of study/type of scholar 

2005-06 

2006-07 

2007-08 

2008-09 

Scholars who earned the primary degree 

2.67 

4.56 

4.44 

11.33 

Scholars who earned the primary credential 

3.17 

6.50 

7.33 

13.08 

Related Services (n=6) 





Total number of scholars enrolled 

190 

232 

266 

303 

Mean number of scholars per course of study 

17.27 

21.09 

24.18 

27.55 

Full-time scholars enrolled 

11.36 

15.36 

18.18 

20.73 

Part-time scholars enrolled 

5.91 

5.73 

6.00 

6.82 

Scholars new to the course of study 

7.45 

12.36 

13.64 

15.18 

Scholars who earned the primary degree 

8.50 

11.10 

11.30 

13.90 

Scholars who earned the primary credential 

7.40 

9.00 

9.90 

11.10 

Minority Institutions (n=l 1) 





Total number of scholars enrolled 

47 

91 

191 

306 

Mean number of scholars per course of study 

4.27 

8.27 

17.36 

27.82 

Full-time scholars enrolled 

3.82 

5.45 

9.18 

18.27 

Part-time scholars enrolled 

0.45 

2.82 

8.18 

9.55 

Scholars new to the course of study 

1.45 

6.00 

11.36 

15.73 

Scholars who earned the primary degree 

2.25 

2.13 

6.13 

8.00 

Scholars who earned the primary credential 

2.18 

2.27 

4.91 

8.27 


TABLE READS: Among 80 FY 2006 funded predominant courses of study, the total number of scholars enrolled in 2005- 
06 was 490 (mean of 6.13 per course of study); the number enrolled in 2006-07 was 877 (mean of 10.96); the number 
enrolled in 2007-08 was 1,334 (mean of 16.68); and the number enrolled in 2008-09 was 1,607 (mean of 20.09). 

NOTE: This table provides 4 years of enrollment and completion data for scholars in FY 2006 funded courses of study: the 
academic year before grant funding (2005-06), a transitional year (2006-07) in which grant activities began, and two 
academic years (2007-08 and 2008-09) when the funded courses of study fully used their grant funding. The number of 
scholars in each category was counted as zero for courses of study that did not exist in a particular year. Calculations for this 
table are based only on data from the predominant course of study within each grant. Aggregate data were provided by 
project directors on all scholars enrolled in a course of study, not only those scholars who received direct financial support 
from the PDP grant. One FY 2006 funded course of study did not complete relevant data for survey item 20 and therefore is 
not included in analyses for this table. Denominators for individual rows within priority/focus areas may vary based on 
course of study characteristics, including provision of degrees and credentials. 

SOURCE: THE Course of Study Survey - Item 20 (appendix D). 
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Table 6.15: Total and mean enrollment and completion among scholars in FY 2007 funded 
predominant courses of study, by PDP priority/focus area: Academic years 
2006-07 through 2008-09 


Academic year 


Course of study/type of scholar 

Prior to 
grant 

2006-07 

Grant 

activities 

begin 

2007-08 

Grant fully 
utilized 

2008-09 

All FY07 funded courses of study, all priority/focus areas (n=100) 



Total number of scholars enrolled 

1,351 

1,964 

2,592 

Mean number of scholars per course of study 

13.51 

19.64 

25.92 

Full-time scholars enrolled 

9.98 

12.73 

15.79 

Part-time scholars enrolled 

3.67 

6.91 

10.13 

Scholars new to the course of study 

6.10 

11.03 

11.41 

Scholars who earned the primary degree 

5.23 

4.78 

8.34 

Scholars who earned the primary credential 

7.34 

7.25 

11.29 

Leadership (n=21) 




Total number of scholars enrolled 

25 

112 

147 

Mean number of scholars per course of study 

1.19 

5.33 

7.00 

Full-time scholars enrolled 

1.19 

4.43 

6.14 

Part-time scholars enrolled 

0 

0.90 

0.86 

Scholars new to the course of study 

0.29 

3.86 

2.90 

Scholars who earned the primary degree 

0.60 

0.25 

0.35 

Scholars who earned the primary credential 

0 

0 

1.33 

Early Childhood (n=14) 




Total number of scholars enrolled 

47 

161 

281 

Mean number of scholars per course of study 

3.36 

11.50 

20.07 

Full-time scholars enrolled 

1.29 

3.07 

7.50 

Part-time scholars enrolled 

2.07 

8.43 

12.57 

Scholars new to the course of study 

1.93 

10.86 

11.21 

Scholars who earned the primary degree 

1.93 

3.07 

8.36 

Scholars who earned the primary credential 

1.77 

2.62 

8.62 

Low Incidence (n=16) 




Total number of scholars enrolled 

193 

285 

384 

Mean number of scholars per course of study 

12.06 

17.81 

24.00 

Full-time scholars enrolled 

2.69 

4.88 

7.75 

Part-time scholars enrolled 

9.38 

12.94 

16.25 

Scholars new to the course of study 

5.50 

11.19 

12.44 

Scholars who earned the primary degree 

4.80 

4.70 

6.90 

Scholars who earned the primary credential 

4.56 

4.44 

6.38 

Preservice Improvement (High Incidence FY 2007) (n=20) 




Total number of scholars enrolled 

891 

1,012 

1,294 

Mean number of scholars per course of study 

44.55 

50.60 

64.70 

Full-time scholars enrolled 

38.32 

39.85 

44.75 

Part-time scholars enrolled 

8.58 

10.75 

19.95 
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Table 6.15: Total and mean enrollment and completion among scholars in FY 2007 funded 
predominant courses of study, by PDP priority/focus area: Academic years 
2006-07 through 2008-09 (continued) 


Course of study/type of scholar 

Prior to 
grant 
2006-07 

Academic year 
Grant 
activities 
begin 
2007-08 

Grant fully 
utilized 
2008-09 

Scholars new to the course of study 

21.26 

22.58 

27.47 

Scholars who earned the primary degree 

18.69 

13.00 

20.71 

Scholars who earned the primary credential 

21.15 

16.90 

23.50 

Related Services (n=14) 

Total number of scholars enrolled 

99 

157 

190 

Mean number of scholars per course of study 

7.07 

11.21 

13.57 

Full-time scholars enrolled 

6.86 

11.07 

13.29 

Part-time scholars enrolled 

0.21 

0.14 

0.29 

Scholars new to the course of study 

3.36 

6.86 

7.57 

Scholars who earned the primary degree 

3.50 

5.57 

6.85 

Scholars who earned the primary credential 

3.43 

5.92 

6.62 

Minority Institutions (n=15) 

Total number of scholars enrolled 

96 

237 

296 

Mean number of scholars per course of study 

6.40 

15.80 

19.73 

Full-time scholars enrolled 

5.20 

7.13 

9.33 

Part-time scholars enrolled 

1.20 

8.67 

10.40 

Scholars new to the course of study 

1.86 

10.33 

5.67 

Scholars who earned the primary degree 

1.54 

2.69 

7.08 

Scholars who earned the primary credential 

1.43 

4.00 

8.43 


TABLE READS: Among 100 FY 2007 funded predominant courses of study, the total number of scholars enrolled in 2006- 
07 was 1,351 (mean of 13.51 per course of study); the number enrolled in 2007-08 was 1,964 (mean of 19.64); and the 
number enrolled in 2008-09 was 2,592 (mean of 25.92). 

NOTE: This table shows enrollment and completion data for scholars in FY 2007 funded courses of study for 3 years: the 
academic year before grant funding (2006-07), a transitional year (2007-08), and one academic year when the funded 
courses of study were operating with their foil grant funding (2008-09). The number of scholar in each category was 
counted as zero for courses of study that did not exist in a particular year. Calculations for this table are based only on data 
from the predominant course of study within each grant. Aggregate data were provided by project directors on all scholars 
enrolled in a course of study, not only those scholars who received direct financial support from the PDP grant. One funded 
course of study in the Preservice Improvement (High Incidence FY 2007) priority area had not yet been developed by 
academic year 2008-09. Four FY 2007 funded courses of study did not complete relevant data for survey item 20. Data from 
these courses of study are therefore not included in analyses for this table. Denominators for individual rows within 
priority/focus areas may vary based on course of study characteristics, including provision of degrees and credentials. 

SOURCE: IHE Course of Study Survey - Item 20 (appendix D). 
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Scholar dropout among funded courses of study 

In AY 2008-09, across all priority/focus areas in courses of study funded in the FY 2006 
or FY 2007 competitions, 174 (4 percent) of 4,199 scholars dropped out. These dropouts 
represented an overall average of approximately one scholar per course of study (table 6.16). 
Thirty-eight percent of courses of study reported at least one scholar leaving in 2008-09 without 
completing the course of study. 


Table 6.16: Dropout among scholars in predominant courses of study funded in FY 2006 
or FY 2007, by PDP priority/focus area: Academic year 2008-09 


Scholars who left in 2008-09 without 
completing the course of study 




Mean number 

Percent of 



per course of 

funded courses 

Number of 

Percent of 

study of 

of study that had 

scholars 

scholars 

scholars 

one or more 

who left 

who left 

who left 

scholars leave 


All priority/ focus areas (n=181) 

174 

4 

0.96 

38 

Leadership (n=43) 

21 

5 

0.49 

30 

Early Childhood (n=24) 

15 

3 

0.63 

29 

Low Incidence (n=30) 

61 

9 

2.03 

53 

High Incidence (FY 2006) 
(n=12) 

8 

3 

0.67 

25 

Preservice Improvement 
(High Incidence FY 2007) (n=20) 

32 

2 

1.60 

50 

Related Services (n=26) 

6 

1 

0.23 

23 

Minority Institutions (n=26) 

31 

5 

1.19 

54 


TABLE READS: Among 181 predominant courses of study funded in FY 2006 or FY 2007, 174 scholars (or4 percent) left in 
2008-09 without completing the course of study, which is a mean of 0.96 scholars per course of study. Thirty-eight percent of 
courses of study reported scholars leaving in 2008-09 without completing the course of study. 

NOTE: Not included in the table are four courses of study that did not complete the relevant survey item, and one course of study 
funded under the Preservice Improvement (High Incidence FY 2007) priority area that had not been developed by academic year 
2008-09. 

SOURCE: IHE Course of Study Survey - Items 20 and 26 (appendix D), administered in fall 2009 through winter 2010. 
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Characteristics of course of study completers 

Across all priority/focus areas, between the year of grant funding and AY 2008-09, 3,038 
scholars completed his or her course of study, averaging slightly more than 17 scholars per 
course of study (table 6.17). Forty-six percent of the total course of study completers earned 
master’s or education specialist degrees, 15 percent earned associate’s or bachelor’s degrees, and 
4 percent earned doctor’s degrees. Within courses of study supported by Leadership grants, 
doctor’s degrees were predominant (81 percent of completers), and for courses of study 
supported by Preservice Improvement grants, associate’s or bachelor’s degrees were the most 
common (37 percent of completers). Eighty-six percent of completers across all priority/focus 
areas obtained state issued credentials. 

The roles for which completers were prepared were generally consistent with the 
priority/focus area that supported their course of study. Among courses of study supported by 
High Incidence and Preservice Improvement grants, 100 percent of completers were prepared as 
special education teachers. Within courses of study supported by Related Services grants, 
speech-language pathologists were most commonly prepared (82 percent). Courses of study 
supported by Minority Institutions grants predominantly prepared special education teachers (70 
percent), but also prepared scholars for a variety of other roles. Across all priority/focus areas, 42 
percent of completers were prepared to serve students with high incidence disabilities, and 24 
percent were trained to work with all disabilities. The most common age range completers were 
prepared to serve was general school age (46 percent), followed by birth to 2 1 or an unspecified 
age (24 percent). 


114 



Table 6.17: Characteristics of completers of FY 2006 and FY 2007 funded predominant courses of study between the year of 
grant funding and the 2008-09 year, by PDP priority/focus area 



All 

priority/ 
focus areas 

Early 

Leadership Childhood 

Low 

Incidence 

High 
Incidence 
(FY 2006) 

Preservice 
Improvement 
(High Incidence 
FY 2007) 

Related 

Services 

Minority 

Institutions 


(n=175) 

(n=42) 

(n=24) 

(n=30) 

(n=12) 

(n=18) 

(n=24) 

(n=25) 

Number of completers 1 
Mean number of 

3,038 

89 

366 

461 

324 

900 

539 

359 

completers per course 
of study 

17.36 

2.12 

15.25 

15.37 

27.00 

50.00 

22.46 

14.36 

Percent of completers obtaining particular primary degree 







Associate’s or 

15 

0 

A 

13 

/T 

37 

0 

7 

bachelor’s degree 

4 

0 

Master’s or 
educational 
specialist degree 

46 

11 

70 

40 

51 

8 

97 

55 

Doctor’s 

4 

81 

0 

0 

0 

0 

1 

10 

Percent of completers obtaining particular primary credentials 2 






State issued 
credentials 

86 

4 

87 

77 

100 

90 

83 

96 

Professional 

organization 

credentials 

18 

0 

0 

0 

0 

0 

83 

23 

Grantee issued 
endorsement or 

'X 

3 

A 

0 

0 

9 

1 

0 

course completion 


4 

o 

A 


only 



Table 6.17: Characteristics of completers of FY 2006 and FY 2007 funded predominant courses of study between the year of 
grant funding and the 2008-09 year, by PDP priority/focus area (continued) 



All 

priority/ 
focus areas 
(n=175) 

Early 

Leadership Childhood 

(n=42) (n=24) 

Low 

Incidence 

(n=30) 

High 
Incidence 
(FY 2006) 
(n=12) 

Preservice 
Improvement 
(High Incidence 
FY 2007) 
(n=18) 

Related 

Services 

(n=24) 

Minority 

Institutions 

(n=25) 

Percent of completers prepared for particular role 







Dually certified 
general/special 
educators 

# 

0 

0 

0 

0 

0 

0 

3 

Early intervention 
or early childhood 
specialist 

11 

0 

89 

2 

0 

0 

0 

1 

Leadership 

personnel 

3 

100 

0 

0 

0 

0 

0 

0 

Psychologists 

1 

0 

0 

0 

0 

0 

2 

10 

Related Services 
personnel 
(excluding speech- 
language) 

5 

0 

0 

10 

0 

0 

17 

1 

Special education 
teachers 

63 

0 

11 

88 

100 

100 

0 

70 

Speech-language 

pathologists 

16 

0 

0 

0 

0 

0 

82 

15 

Percent of completers prepared to serve 

particular disability 

areas 3 






All disabilities 

24 

82 

83 

0 

0 

0 

19 

70 

High incidence 

42 

1 

0 

0 

89 

100 

2 

17 

Low incidence 

9 

0 

0 

48 

0 

0 

2 

8 

Autism 

5 

2 

6 

19 

0 

0 

0 

11 



Table 6.17: Characteristics of completers of FY 2006 and FY 2007 funded predominant courses of study between the year of 
grant funding and the 2008-09 year, by PDP priority/focus area (continued) 



All 
priority/ 
focus areas 
(n=175) 

Leadership 

(n=42) 

Early 

Childhood 

(n=24) 

Low 

Incidence 

(n=30) 

High 
Incidence 
(FY 2006) 
(n— 1 2) 

Preservice 
Improvement 
(High Incidence 
FY 2007) 
(n=18) 

Related 

Services 

(n=24) 

Minority 

Institutions 

(n=25) 

Deaf-blindness 

1 

0 

0 

2 

0 

0 

0 

3 

Developmental 

delay 

# 

0 

2 

0 

0 

0 

0 

0 

Emotional 

disturbance 

1 

3 

0 

0 

11 

0 

0 

0 

Hearing impairment 

5 

0 

17 

10 

0 

0 

4 

5 

Mental retardation 

2 

6 

0 

6 

0 

0 

0 

4 

Other health 
impairment, 
including attention 
deficit disorder 

# 

1 

0 

0 

0 

0 

0 

0 

Speech-language 

impairment 

14 

4 

0 

0 

0 

0 

72 

6 

Specific learning 
disabilities 

1 

0 

0 

0 

5 

0 

5 

0 

Traumatic brain 
injury 

# 

0 

0 

0 

0 

0 

1 

0 

Visual impairment 

4 

0 

4 

21 

0 

0 

0 

0 

Percent of completers prepared to serve 

particular grade/age levels 4 






Birth to pre-school 

14 

12 

100 

2 

0 

0 

4 

7 

Elementary school 

13 

0 

13 

0 

29 

18 

0 

22 

Secondary school 

8 

0 

0 

4 

13 

18 

0 

7 



Table 6.17: Characteristics of completers of FY 2006 and FY 2007 funded predominant courses of study between the year of 
grant funding and the 2008-09 year, by PDP priority/focus area (continued) 



All 

priority/ 
focus areas 
(n=175) 

Leadership 

(n=42) 

Early 

Childhood 

(n=24) 

Low 

Incidence 

(n=30) 

High 
Incidence 
(FY 2006) 
(n=12) 

Preservice 
Improvement 
(High Incidence 
FY 2007) 
(n=18) 

Related 

Services 

(n=24) 

Minority 

Institutions 

(n=25) 

Transition or post- 
secondary 

3 

0 

0 

22 

0 

0 

1 

0 

Birth to 21 (or 
unspecified age) 

24 

31 

0 

26 

5 

0 

90 

26 

General school age 

46 

56 

0 

54 

54 

82 

6 

46 


TABLE READS: Among 175 predominant courses of study funded through FY 2006 and FY 2007 competitions, there were 3,038 completers between the year of grant 
funding and the 2008-09 year. Of these completers, 15 percent earned associate’s or bachelor’s degrees, 46 percent earned master’s or educational specialist degrees, and 4 
percent earned doctor’s degrees. 

* Rounds to zero. 

’Total number of completers is based on the number of scholars earning the primary degree and/or the primary credential. 

2 Select courses of study identified more than one type of certification provided. Therefore, completers may be counted in more than one certification area. 

3 Select courses of study identified more than one disability area for which completers were prepared. Therefore, completers may be counted in more than one disability area. 
4 Select courses of study identified more than one age/grade level for which completers were prepared. Therefore, completers may be counted in more than one age/grade 
level. 

NOTE: IE1E Course of Study Survey data were reported at the course of study level, rather than the individual scholar level. Therefore, data related to completers of PDP- 
funded courses of study may include scholars not receiving stipends or other monetary support related to the PDP grant providing funding for a given course of study. One 
funded course of study in the Preservice Improvement (High Incidence FY 2007) priority area had not yet been developed by academic year 2008-09, while 10 other funded 
courses of study did not complete relevant survey items for this table. Data from these courses of study are therefore not included in analyses for this table. 

SOURCE: IHE Course of Study Survey - Items 5, 6, and 20 (appendix D). 



What were the quality and relevance/usefulness of new or significantly 
modified components for funded courses of study? 

In the summer of 2010, expert panel members reviewed a sample of 134 new or 
significantly modified components from 99 courses of study funded with PDP training grants in 
FY 2006 or FY 2007. The components had been developed or modified since the time of grant 
application. Experts rated all 134 components for quality and 70 components for relevance/ 
usefulness on a 5-point scale, with 5 being the highest. 16 Three panelists rated each component 
(figure 6.2). 

Figure 6.2: Distribution of expert panelist ratings (on a 5-point scale, with 5 being the 

highest) of quality and relevance/usefulness of new or significantly modified 
course of study components funded with FY 2006 or FY 2007 PDP training 
grants 

Percent of ratings 



(n-134 course of study components, (n=70 course of study components, with 
with 2,604 quality ratings) 630 relevance/usefulness ratings) 


Very Low 
Low 

■ Moderate 

■ High 

■ Very High 


FIGURE READS: For the 134 new or significantly modified course of study components reviewed by an expert panel for quality, 5 
percent of the 2,604 ratings were at the level of 1 (very low); 1 1 percent were 2 (low); 26 percent were 3 (moderate); 25 percent were 4 
(high); and 33 percent were 5 (very high). The mean quality rating was 3.71 . 

NOTE: The data that are summarized in this figure are weighted using the inverse of the probability of selection for each sampled 
change component to allow generalization to all change components reported by grantees. Relevance/usefulness composites are based on 
ratings from Courses or Classes and Training Units or Modules components only. 

SOURCE: Expert panel reviews, conducted July through October 2010. 


16 The rubrics developed to evaluate each type of course of study component (appendix E) originally distinguished some 
indicators as measures of quality, and other indicators as measures of relevance/usefulness. However, during testing of the 
rubrics, expert consultants recommended that it would be more appropriate to classify all of the indicators for components 
involving new faculty, fieldwork, and mentoring programs as quality indicators. 
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Fifty-eight percent of the quality ratings and 57 percent of the relevance/usefulness ratings were 
high or very high. Mean ratings were 3.71 for quality and 3.63 for relevance/usefulness (table 
6.18). Table 6.18 provides additional details regarding the number of indicators, number of 
ratings, and scores for the specific types of components that were reviewed. 


Table 6.18: Mean expert panelist ratings (on a 5-point scale, with 5 being the highest) of a 
sample of new and significantly modified course of study components, by 
review dimension, and by PDP priority/focus area and component type 




Quality 


Relevance/usefulness 1 



Number of 
components 

Number of 
ratings 

Mean 

SD 

Number of 
components 

Number of 
ratings 

Mean 

SD 

All components combined 

134 

2,604 

3.71 

2.17 

70 

630 

3.63 

1.73 

By PDP priority/focus Area 









Leadership 

29 

528 

3.86 

1.97 

19 

171 

3.48 

1.41 

Early Childhood 

20 

405 

3.53 

1.76 

9 

81 

3.07 

0.81 

Low Incidence 

20 

423 

3.62 

2.31 

9 

81 

3.37 

1.99 

High Incidence (FY 2006) 

5 

114 

3.61 

1.63 

1 

9 

4.33 

t 

Preservice Improvement 
(High Incidence FY 2007) 

18 

372 

3.96 

2.53 

8 

72 

4.17 

1.99 

Related Services 

15 

306 

3.65 

1.98 

7 

63 

3.16 

1.59 

Minority Institutions 

27 

456 

3.45 

2.35 

17 

153 

3.66 

1.64 

By component type 









Courses or classes 

42 

756 

3.75 

3.25 

42 

378 

3.65 

2.10 

Training units or modules 

28 

336 

3.71 

1.14 

28 

252 

3.37 

0.93 

New faculty 

17 

357 

3.55 

1.27 

t 

t 

t 

t 

Fieldwork 

28 

756 

3.70 

1.77 

t 

t 

t 

t 

Mentoring programs 

19 

399 

3.36 

1.46 

t 

t 

t 

t 


TABLE READS: The 134 change components reviewed by the expert panels for quality received 2,604 ratings, and the 
70 change components reviewed by the expert panels for relevance/usefulness received 630 ratings; the overall mean 
rating for quality was 3.71 ( sd=2.17 ), and the overall mean rating for relevance/usefulness was 3.63 ( sd=l . 73). 

fNot Applicable. Rubrics for new faculty, fieldwork, and mentoring programs contained only the Quality dimension. 

l 

Rclevance/usefulness composites are based on ratings from courses or classes and training units or modules 
components types only. 

NOTE: Overall, experts made 3,234 ratings across the 134 components. All ratings were on a 5-point scale, with 5 as the 
highest value. Data in this table are weighted using the inverse of the probability of selection for each sampled change 
component to allow generalization to all change components reported by grantees. 

SOURCE: Expert panel reviews, conducted July through October 2010 using the rubric in appendix E. 
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What became of courses of study that did not receive PDP training grant 
funding? 

Of the 537 applications from IHEs to the FY 2006 and FY 2007 PDP training grant 
competitions, 347 were not funded. Six nonfunded applications were removed from the sample 
because we were unable to identify either the project director or the course of study within the 
application due to missing information. The remaining 341 nonfunded applications formed the 
nonfunded sample. Where extant data were used, all 341 nonfunded applications are described. 
Ninety FY 2006 applications were requests for funding for a course of study that was also the 
focus of a subsequent application and were removed from the sample, and two applications were 
used for survey piloting. Of the 249 nonfunded applications in the final survey sample, responses 
were obtained for 230. Analyses involving survey data are based on these 230 nonfunded 
applications. 

Across the 341 nonfunded PDP applications from FY 2006 and FY 2007, 10 percent 
contained multiple courses of study. When the term “course of study” is used throughout the 
sections that follow, it is in reference to the predominant course of study for which the most 
funding was requested within an application. 

Development, maintenance, and implementation of nonfunded courses of study 

Among all the 230 nonfunded predominant courses of study for which survey data were 
gathered, 34 percent were developed or maintained without PDP funding (table 6.19). Among 
the 156 nonfunded courses of study that were intended to be substantially different from any 
existing course of study at the submitting IHE, 8 percent were developed. Of the 74 nonfunded 
existing courses of study that were intended to be maintained or enhanced through the PDP 
request, 88 percent were maintained without the PDP funding. Most commonly, nonfunded 
courses of study that were developed or maintained without PDP funding did so without other 
external funding. 

Subsequent funding status of nonfunded applicants 

Of 1 14 IHEs with nonfunded applications in FY 2006, 70 percent reapplied to the same 
priority/focus area within 4 years, and 50 percent received funding in that time period (table 
6.20). For FY 2007, 56 percent of 108 IHEs with nonfunded applications reapplied to the same 
priority/focus area within 3 years, and 33 percent were funded. IHEs with Feadership 
applications most commonly reapplied: 89 percent of FY 2006 nonfunded applicants reapplied 
within 4 years, and 62 percent were funded; and 83 percent of FY 2007 nonfunded applicants 
reapplied within 3 years, and 54 percent were funded. 
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Table 6.19: Percent of predominant courses of study proposed but not funded in FY 2006 or FY 2007 that were developed or 
maintained without PDP funding and the extent of external funding used, by relationship to existing courses of 
study and by PDP priority/focus area 



All 





Preservice 





Early 


High 

Improvement 

Related 

Minority 


priority/ 


Low 

Incidence 

(High Incidence 


focus areas 

Leadership 

Childhood 

Incidence 

(FY 2006) 

FY 2007) 

Services 

Institutions 

All nonfunded courses of study 









(n=230) 

(n=63) 

(n=16) 

(n=28) 

(n=29) 

(n=14) 

(n=32) 

(n=48) 

Developed or 
maintained without 
PDP funding 

34 

33 

44 

39 

52 

43 

25 

21 

With external funding 
equal to or greater than 
PDP requested amount 

1 

2 

0 

4 

0 

0 

0 

0 

With external funding 
smaller than PDP 

6 

8 

0 

14 

10 

0 

3 

0 

requested amount 

Without external 
funding 

23 

19 

44 

14 

38 

36 

22 

17 

Source or amount of 

A 

C 

0 

7 

7 

7 

0 

A 

funding not identified 

4 

D 

/ 

D 

/ 

4 

Nonfunded courses of study that were substantially different from any existing course of study 




(n=156) 

(n=45) 

(n=10) 

(n=18) 

(n=14) 

(n=9) 

(n=24) 

(n=36) 

Developed without 
PDP funding 

8 

7 

10 

17 

14 

11 

4 

6 

With external funding 
equal to or greater than 
PDP requested amount 

0 

0 

0 

0 

0 

0 

0 

0 

With external funding 
smaller than PDP 
requested amount 

1 

2 

0 

0 

7 

0 

0 

0 



Table 6.19: Percent of predominant courses of study proposed but not funded in FY 2006 or FY 2007 that were developed or 
maintained without PDP funding and the extent of external funding used, by relationship to existing courses of 
study and by PDP priority/focus area (continued) 



All 
priority/ 
focus areas 

Leadership 

Early 

Childhood 

Low 

Incidence 

High 
Incidence 
(FY 2006) 

Preservice 
Improvement 
(High Incidence 
FY 2007) 

Related 

Services 

Minority 

Institutions 

Without external 

c 

9 

10 

11 

7 

11 

A 


funding 


z 



■D 

Source or amount of 

9 

9 

0 

f. 

0 

0 

0 


funding not identified 

L 

Z 

O 


Nonfunded courses of study that were maintained from existing courses of study 






(n=74) 

(n=18) 

(n=6) 

(n=10) 

(n=15) 

(n=5) 

(n=8) 

(n=12) 

Courses of study 
maintained 

88 

100 

100 

80 

87 

100 

88 

67 

With external funding 
equal to or greater than 
PDP requested amount 

3 

6 

0 

10 

0 

0 

0 

0 

With external funding 
smaller than PDP 

15 

22 

0 

40 

13 

0 

12 

0 

requested amount 

Without external 
funding 

62 

61 

100 

20 

67 

80 

75 

58 

Source or amount of 
funding not identified 

8 

11 

0 

10 

7 

20 

0 

8 


TABLE READS: Among the 230 predominant courses of study that were proposed but not funded in either FY 2006 or FY 2007, 34 percent were developed or maintained 
without PDP funding. One percent of courses of study were developed or maintained through non-PDP funding sources equal to or greater than the PDP requested amount, 6 
percent were developed or maintained through non-PDP funding sources smaller than the PDP requested amount, 23 percent were developed or maintained without any 
external funding, and 4 percent were developed or maintained though the source and amount of funding was not identified. 

SOURCE: IHE Course of Study Survey - Items 1, 2, and 3 (appendix D), administered in fall 2009 through winter 2010. 



Table 6.20: Funding status of nonfunded applicants reapplying for funding in the same priority/focus area, by PDP 
competition year and priority/focus area 



Percent of IHEs that reapplied to the same 


Percent of IHEs that were subsequently funded in the 



PDP priority/focus 

area 



same PDP priority/focus 

area 



Reapplied 











by 

FY 

FY 

FY 

FY 

Funded by 

FY 

FY 

FY 

FY 


FY 2010 

2007 

2008 

2009 

2010 

FY 2010 

2007 

2008 

2009 

2010 

IHEs with 
nonfunded 
applications in 

70 

54 

36 

29 

32 

50 

28 

21 

17 

11 

FY 2006 (n=114) 

Leadership 

(n=45) 

89 

76 

60 

51 

44 

62 

33 

20 

24 

18 

Early Childhood 
(n=10) 

60 

50 

0 

30 

30 

30 

30 

0 

10 

10 

Low Incidence 
(n=12) 

83 

75 

42 

25 

42 

42 

17 

25 

25 

0 

High Incidence 1 
(n=37) 

35 

22 

8 

3 

8 

19 

5 

8 

0 

5 

Related Services 
(n=27) 

74 

48 

37 

26 

19 

63 

33 

37 

11 

4 

Minority Institutions 
(n=25) 

68 

52 

12 

12 

16 

40 

28 

8 

8 

0 


IHEs with 
nonfunded 
applications in 
FY 2007 (n=108) 

56 

t 

37 

32 

31 

33 

t 

19 

17 

Leadership 

(n=41) 

83 

t 

66 

59 

41 

54 

t 

29 

27 

Early Childhood 

(n=ll) 

55 

t 

9 

45 

9 

18 

t 

0 

18 

Low Incidence 
(n=22) 

59 

t 

32 

23 

36 

27 

t 

14 

14 


5 



Table 6.20: Funding status of nonfunded applicants reapplying for funding in the same priority/focus area, by PDP 
competition year and priority/focus area (continued) 



Percent of IHEs that reapplied to the same 
PDP priority/focus area 


Percent of IHEs that were subsequently funded in the 
same PDP priority/focus area 


Reapplied 

by 

FY 2010 

FY 

2007 

FY 

2008 

FY 

2009 

FY 

2010 

Funded by 
FY 2010 

FY 

2007 

FY 

2008 

FY 

2009 

FY 

2010 

Preservice 

Improvement 

14 

t 

7 

0 

7 

7 

t 

7 

0 

0 

(n=14) 

Related Services 
(n=15) 

67 

t 

40 

20 

27 

47 

t 

40 

13 

0 

Minority Institutions 

32 

t 

12 

9 

18 

15 

t 

3 

6 

6 

(n=34) 










TABLE READS: Among the 1 14 institutions of higher education (IHEs) that submitted applications in FY 2006 that were not funded, 70 percent reapplied to the same PDP 
priority/focus area within 4 years, and 50 percent were funded in those competitions; 54 percent reapplied to the same PDP priority/focus area in FY 2007 competitions, and 28 
percent were funded; 36 percent reapplied to the same PDP priority/focus area in FY 2008 competitions, and 21 percent were funded; 29 percent reapplied to the same PDP 
priority/focus area in FY 2009 competitions, and 17 percent were funded; and 32 percent reapplied to the same PDP priority/focus area in FY 2010 competitions, and 1 1 
percent were funded. 

fNot applicable. FY 2007 data are not relevant for IHEs with FY 2007 nonfunded applications. 

1 Nonfunded applications submitted to the FY 2006 High Incidence focus area are matched for the purposes of this analysis to applications submitted to the Preservice Training 
Improvement priority area in subsequent years. 

NOTE: This analysis focused at the IHE level, not the course of study level, because information on subsequent applications was not available at the course of study level. 
Thus, subsequent applications, although from the same IHE and to the same PDP priority/focus area, were not necessarily for the same course of study that was the focus of 
the FY 2006 or FY 2007 nonfunded application. 

SOURCE: Office of Special Education Programs PDP application slates for fiscal years 2006 through 2010. 
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Appendix A 


Semi-Structured Interview Protocol for Center Director 

[Introduction includes a description of the evaluation, explanation of the goals of the interview, and 
information for interviewee on what the upcoming questions will be about.] 

Name of Interviewee 

Date Began as Center Director 

Part I. Center Goals and Objectives 

We have reviewed your grant proposal, progress reports, and your inventory, but we realize that the focus 
of activities often changes over time, so we want to be sure we have a good picture of what the center’s 
focus has been up to this point. 

1. Please start by giving us a description of the center’s goals and objectives. 

2. In what ways is the work of the center a continuation of work that the center has done previously? 

3. Has the work of the center changed since you were initially funded? [If yes,] What prompted that 
change? 

4. Are there any obstacles and challenges that the center has faced in completing the work that you 
envisioned? 

5. [If the grant is ongoing] What do you envision as the future direction of the center? 

6. [If the grant is over] In what ways has the center’s work been sustained since the end of the grant? 

7. To your knowledge, which center products, services, or activities have been used the least 
extensively? 

8. To your knowledge, which center products, services, or activities have been used the most 
extensively? 


Part II. Organization Structure and Staffing 

1. Please give me an overview of the center’s organizational structure. 

2. Where is the center housed (e.g., a department within the university)? 

3. How many people work at the center and what are their roles? 

4. Does the center use any subcontractors or outside consultants? [If yes,] What are their roles? 

Part III. Inventory of Products and Services [In this part of the interview, ask interviewee to clarify 
information on the inventory if needed.] 


A-l 



Appendix B 


PDP National Centers Inventory of Products and Services 


Dear Center Director, 

Enclosed are a project inventory fonn and instructions to be used to compile a 
complete list of your center’s products and services during its funding from the U.S. 
Department of Education’s Personnel Preparation Program to Improve Services and 
Results for Children with Disabilities. 

The evaluation will use the completed product and service inventory to select a 
sample of products and services for review by an expert panel. Please designate up to 
10% (or up to 3 if fewer than 30 items are listed) products or services that you believe 
best represent the work of your center; these products will be included among the sample 
of products to be reviewed by the expert panel. This panel will rate nominated and 
sampled products on several dimensions, such as quality, relevance, and usefulness. 

Within 10 days after receiving this inventory, a member of the study team will contact 
you to see if you have any questions about how to complete the inventory. We ask that 
you complete the inventory by Januaryl5, 2010 and return it to elainecarlson@westat.com . If 
you prefer to send paper, please mail the completed inventory to: 

Elaine Carlson 
Westat 

1600 Research Blvd., RA1219 
Rockville, MD 20850 

Once we receive the completed inventory, a member of the study team will contact you within 
one month to discuss the list of products/services you have submitted and to ask you a few more 
questions about your center, and the product and services you have developed. 

Thank you for support of the evaluation of the Personnel Preparation Program to 
Improve Services and Results for Children with Disabilities . 


For this data collection, Westat will report results for each Center. Data about your Center will be available to 
the Department of Education and will appear in evaluation reports. 
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Instructions for Completing the Inventory 


I. Background 

The U.S. Department of Education’s (ED) National Center for Education Evaluation is 
conducting a study of the Personnel Preparation Program to Improve Services and Results for 
Children with Disabilities. All National Centers funded between 2001 and 2007 are being asked 
to participate in this important evaluation being carried out by Westat and its subcontractors, the 
Council for Exceptional Children and Compass Consulting. For the evaluation, we will select a 
sample of products and services developed by your center for expert review, including several 
products or services nominated by you as your center’s “signature works.” 

II. Identifying Products and Services for Inclusion in the Inventory 

The unit of analysis for the expert review panels will be the product or service. For the 
purposes of this inventory, a “product” or “service” will ordinarily comprise a group of closely 
related activities designed to achieve a specific outcome for a specific audience. The 
inventory should include all of the products or services developed or delivered by your National 
Center, including those developed or delivered in collaboration with other entities, during the 
grant period. It is critical that the inventory reflect all the products and services attributable to 
your center during the grant period. 

Each product or service entered in the inventory should be a complete and coherent 
whole. Because each product or service listed in the inventory could potentially be sampled for 
expert panel review, each product or service shown as an entry (or row) in the inventory form 
should be complete and coherent — that is, works that can be understood and rated on their own 
by expert panel members who may not know anything about other aspects of the center’s work. 

Avoid listing products or services separately in the inventory if they cannot be understood or 
evaluated without information about related products or services. 

For example, the first example product in the sample inventory form at the end of the 
packet is a training video. This video required extensive planning (e.g., needs assessment, pilot 
testing the video prototype) and follow-up activities (e.g., providing a “helpline” in case users had 
questions about the content of the video). These planning and follow-up activities would be very 
difficult for a panel to rate in the absence of information about the video product itself. 

Therefore, the center should list these planning and follow-up activities and the product itself as a 
single work (one row) on the inventory form. Each phase of the project — planning, the product 
itself, and follow up — should then be described briefly in the “activities and deliverables” 
column. Likewise, in example two, for the Master’s coursework in emotional disturbance, the 
planning activities and all 20 syllabi for the courses should be listed as a single product. 


For this data collection, Westat will report results for each Center. Data about your Center will be available to 
the Department of Education and will appear in evaluation reports. 
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III. What Not to Include in the Inventory 


Products and services that were not intended to be used directly by or while assisting 
entities outside your Center should not be included on the inventory. Such things would include 
but not necessarily be limited to: (1) agendas, notes, or summaries from internal meetings; 

(2) reports or other products (such as annual performance reports) prepared for administrative 
purposes; (3) materials prepared for your own grant evaluator; and (4) general promotional 
materials for your Center. 

IV. Identifying Signature Works 

After listing all your products and services in the inventory, please go back and designate 
which products you consider to be “signature works.” For the purposes of the study, signature 
works are defined as products or services that meet at least of the following criteria: 

• Prominent in terms of the proportion of the National Center’s effort; 

• Exclusively or nearly exclusively a product of the National Center’s efforts; 

• Predominately supported by Personnel Preparation Program grant funds; and 

• Meets a critical need in the field or to the class of customers served. 

V. Completing the Inventory Form 

Centers may use an electronic or paper version of the attached form to complete 
their inventory. You will receive an electronic version of the fonn in Word, after a 
member of the study team has spoken with you to review the form and answer any 
questions you may have. Add rows to the table as needed. 

Column A Number. Assign a sequential number to each product or service. You may want to 
complete this step last, after you have listed all the products/services. 

Column B Signature Work. Indicate signature works by placing an X in this column (up to 
10%, or up to 3 if fewer than 30 items have been listed). 

Column C Name Assign a descriptive name to each product or service. 

Column D. Description - Provide a brief description of the product or service. 

Colu mn E. Medium - Indicate the medium or media, e.g., in-person training, video, research 
paper. 

Column F. Audience or Customer - Indicate the intended audiences or customers for the 
product or service. 


For this data collection, Westat will report results for each Center. Data about your Center will be available to 
the Department of Education and will appear in evaluation reports. 
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Column G. Purpose - Indicate the intended purpose of the product or service. 


Column H. Activities and Deliverables - Provide a description of the activities and deliverables, 
including planning and follow-up activities, as well as the product or service itself. If the product 
or service comprises multiple activities or documents, list them here. For services, include 
location and type(s) of participants. 

Column I. Start Date - Indicate the month and year in which the product or service became 
available. 

Column J. End Date - Indicate the month and year in which the product or service stopped being 
available. Enter ‘ongoing’ if the product or service continues to be available. 

Column K. Estimated Cost - Estimate the cost, in dollars, of developing and delivering the 
product or service. For example, if a particular product or service consumed approximately 50 
percent of the Center’s resources for the first year of the grant, multiply 0.5 by Year 1 
expenditures. Include overhead costs in the estimates. The estimates should be as accurate as 
reasonably possible, but you do not need to complete a detailed audit of costs per product. The 
cost estimates should take no longer than one hour to complete. If costs are ongoing, indicate 
what has been expended to date. If your center (1) disseminated a product developed by another 
individual or institution or (2) developed a product or service in collaboration with another 
individual or institution, include only the costs incurred by your center. The sum of costs for all 
the Center’s products and services should approximate total expenditures to date, excluding costs 
for the Center’s external evaluator. 


For this data collection, Westat will report results for each Center. Data about your Center will be available to 
the Department of Education and will appear in evaluation reports. 
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Products and Services Inventory for National Centers Funded Under the IDEA Personnel Preparation Program 


Center name and grant number: 


A. 

Number 

B. 

Signature 

works 

C. 

Name of 
product 01 - 
service 

D. 

Description 

E. 

Medium 

F. 

Audience/ 

customer 

G. 

Purpose 

H. 

Activities and Deliverables 
(including all the components 
of the product or service) 

I. 

Start 

date 

J. 

End 

date 

K. 

Estimated 

cost 

Ex.#l 


Rtl video 

Technical 
assistance 
training on 
Rtl 

Video, 
DVD, 
and web 

Local 

special 

education 

administra- 

tors 

Provide 

overview of Rtl 
and its role in 
LD 

identification 

Conducted needs assessment 
interviews with 5 school 
districts. Created 15 minute 
video prototype and pilot tested 
with 10 special education 
administrators. 30-minute video 
on different types of Rtl; DVD 
and web on how they can be 
used in LD identification, next 
steps, and where to get 
additional information and 
assistance. Helpline for users 
with follow-up questions about 
applications of training. 

11/06 

Ongoing 

$22,000 


For this data collection, Westat will report results for each Center. Data about your Center will be available to the Department of Education and will appear in 
evaluation reports. 
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A. 

Number 

B. 

Signature 

works 

C. 

Name of 
product 01 - 
service 

D. 

Description 

E. 

Medium 

F. 

Audience/ 

customer 

G. 

Purpose 

H. 

Activities and Deliverables 
(including all the components 
of the product or service) 

I. 

Start 

date 

J. 

End 

date 

K. 

Estimated 

cost 

Ex. #2 


ED 

curriculum 

Curriculum 
for master’s 
degree 
program in 
emotional 
disturbance 

Paper 

University 
administra- 
tors and 
prospective 
students 

Define the 
content of a 60- 
credit graduate 
program for 
preparing 
teachers to serve 
children with 
emotional 
disturbance 

Convened expert panel to 
discuss criteria for Masters 
course in ED. Conducted 
literature review in ED. 
Developed program description 
and 20 course syllabi: the 
Exceptional Learner, 
Contemporary Issues in Special 
Education, Introduction to 
Emotional Disturbance/ 
Behavior Disorders, Classroom 
and Behavior Management, 
Assessing Learners with 
Disabilities, Initial Field 
Experience, Child Growth and 
Development, Adolescent 
Growth and Development, 
Learning Theories in 
Education, Advanced 
Educational Psychology, 
Collaborative Consultation, 
Methods in Adaptive 
Instruction, Intro to Reading 
Difficulties, Teaching 
Remedial Math, Research 
Foundations, Educational 
Research Project, Social 
Foundations in Education, 
Secondary Programming for 
Students with Disabilities, 
Elementary Practicum, 
Middle/Secondary Practicum. 

10/04 

Ongoing 

$5,000 


For this data collection, Westat will report results for each Center. Data about your Center will be available to the Department of Education and will appear in 
evaluation reports. 
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A. 

Number 

B. 

Signature 

works 

C. 

Name of 
product 01 - 
service 

D. 

Description 

E. 

Medium 

F. 

Audience/ 

customer 

G. 

Purpose 

H. 

Activities and Deliverables 
(including all the components 
of the product or service) 

I. 

Start 

date 

J. 

End 

date 

K. 

Estimated 

cost 

























































Col A. Assign a sequential number to each product or service 

Col B. Indicate signature works with X in this column (up to 3, up to 10%) 

Col C. Assign a descriptive name to each product or service 

Col D. Briefly describe the product or service 

Col E. Indicate the media, e.g., in-person training, video, research paper 
Col F. Indicate the intended audiences for the product or service 


Col G. Indicate the intended purpose of the product or service 

Col H. Provide additional description of the product or service 

Col 1. Indicate the date on which the product or service became available 

Col J. Indicate the date on which the product or service stopped being 
available 

Col K. Estimate the cost of developing and delivering the product or service 


For this data collection, Westat will report results for each Center. Data about your Center will be available to the Department of Education and will appear in 
evaluation reports. 




Appendix C 


PDP National Centers Expert Panel Review Rubrics 


CILSPRT 

INDICATORS AND SCORING RUBRIC FOR THE 
CAROLINA INTERDISCIPLINARY LARGE-SCALE POLICY RESEARCH TRAINING 

(CFDA 84.325L) 

Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: PRODUCTS AND SERVICES DEEMED OF HIGH QUALITY: 
Synthesis rating (mean of 1A through 1G) 


Indicator 1A. The course or courses, enrichment activities, and/or experiential learning 
opportunities cover key content areas relevant to conducting large-scale 
analyses in special education. 


Indicator IB. The faculty members or other individuals delivering course content were 
qualified for their assigned role. 


Indicator 1C. The product or service adequately addresses all of the research questions, 
hypotheses, or objectives presented. 


Indicator ID. The product or service’s findings are clearly supported by the data, or, for a 

concept or position paper or presentation, arguments or theories are backed up 
with valid research and sound reasoning. 


Indicator IE. The analyses conducted are appropriate for the data used in the paper or 
presentation. 


Indicator IF. The product or service’s data sources and instruments are appropriate for 
addressing the research questions, hypotheses, or objectives. 


Indicator 1G. The product or service is easy to read and/or understand, given the intended 
audience. 



C-l 


DIMENSION 2: PRODUCTS DEEMED OF HIGH RELEVANCE/USEFULNESS TO 
POLICY OR PRACTICE: Synthesis rating (mean of 2A through 2C) 


Indicator 2A. The paper or course focuses on topical areas relevant to special education 
policy or practice. 


Indicator 2B. The paper or course uses national databases suitable for analyses on children 
with disabilities. 


Indicator 2C. The product or service contributes new information to the intended audience 
on the topic being addressed. 


GENERAL COMMENTS 
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CRITERIA FOR SCORING 


Dimension 1: Products and Services Deemed of High Quality ( Check the number that best describes 
the quality of the product or service) 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1A. The course 

The course or 


The course or 


The course or 


or courses, 

courses, 


courses, 


courses, 


enrichment 

enrichment 


enrichment 


enrichment 


activities, 

activities, 


activities, and/or 


activities, 


and/or 

and/or 


experiential 


and/or 


experiential 

experiential 


learning 


experiential 


learning 

learning 


opportunities 


learning 


opportunities 

opportunities 


cover some 


opportunities 


cover key 

cover key 


content relevant 


cover little if 


content 

content 


to conducting 


any content 


relevant to 

relevant to 


large-scale 


relevant to 


conducting 

conducting 


analyses in 


conducting 


large-scale 

large-scale 


special 


large-scale 


analyses in 

analyses in 


education but 


analyses in 


special 

special 


omit important 


special 


education. 

education. 


content or 
include 
unnecessary 
content. 


education. 



Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IB. The faculty 

The faculty 


The faculty 


The faculty 


members or 

members or 


members or 


members or 


other 

other 


other 


other 


individuals 

individuals 


individuals 


individuals 


delivering 

delivering 


delivering 


delivering 


course 

course 


course content 


course content 


content were 

content were 


were moderately 


were poorly 


qualified for 

highly 


qualified for 


qualified for 


their assigned 

qualified for 


their assigned 


their assigned 


role. 

their assigned 


role. 


role. 



role. 






Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1C. The product or 
service 
adequately 
addresses all of 
the research 
questions, 
hypotheses, or 
objectives 
presented. 

All of the 

research 

questions, 

hypotheses, 

or objectives 

are 

adequately 

addressed. 


Some but 

not all of the 

research 

questions, 

hypotheses, 

or objectives 

are 

adequately 

addressed. 


Few if any 

of the 

research 

questions, 

hypotheses, 

or objectives 

are 

adequately 

addressed. 


Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

ID. The product or 

All of the 


Some but 


Few if any 


service’s 

findings are 


not all of the 


of the 


findings are 

clearly 


findings are 


findings are 


clearly 

supported by 


clearly 


supported by 


supported by 

the data, or, 


supported by 


the data, or, 


the data or, for 

for a concept 


the data, or, 


for a concept 


a concept or 

or position 


for a concept 


or position 


position paper 

paper, all 


or position 


paper, few if 


or presentation. 

arguments or 


paper, some 


any of the 


arguments or 

theories are 


but not all of 


arguments 


theories are 

backed up 


the 


or theories 


backed up with 

with valid 


arguments or 


are backed 


valid research 

research and 


theories are 


up with 


and sound 

sound 


backed up 


valid 


reasoning. 

reasoning. 


with valid 


research or 





research and 


sound 





sound 


reasoning. 





reasoning. 




Comments on rating, required for NA: 


C-5 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IE. The analyses 
conducted are 
appropriate for 
the data used 
in the product 
or service. 

The product or 
service uses 
the best and 
appropriate 
analysis 
methods for 
the research 
questions, 
hypotheses, or 
objectives. 


The analysis 
methods are 
reasonable, 
although 
better ones 
could be 
applied 
within the 
context. 


The product or 
service uses few 
if any 
appropriate 
methods to 
analyze the 
data, or the 
analytic 

methods are not 
described. 


Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IF. The product or 

All of the 


Some but not 


The instruments 


service’s data 

instruments are 


all of the 


have little if any 


sources and 

valid and 


instruments 


validity or 


instruments 

reliable for 


are valid and 


reliability for 


are 

addressing the 


reliable; the 


addressing the 


appropriate for 

research 


validity 


research 


addressing the 

questions, 


and/or 


questions, 


research 

hypotheses, or 


reliability of 


hypotheses, or 


questions, 

objectives, and 


some 


objectives, and 


hypotheses, or 

the data are 


instruments 


there are serious 


objectives. 

collected using 


is in 


problems with 



sound 


question; or 


data collection, 



practices. 


there are 


or 





some 


instrumentation 





concerns 


was applicable 





with the 


to the product 





soundness of 


or service but 





the data 


was not 





collection. 


addressed at all. 



Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1G. The product 

The product 


Some but not 


The product 


or service is 

or service is 


all sections 


or service is 


easy to read 

well written 


of the 


poorly 


and 

throughout 


product or 


written 


understand, 

and can be 


service are 


throughout. 


given the 

easily 


well written. 




intended 

understood 


and/or the 




audience. 

by the 


information 





intended 


is too 





audience. 


technical for 
the intended 
audience. 




Comments on rating, required for NA: 
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Dimension 2: Products Deemed of High Relevance to Policy or Practice ( Check the number that best 
describes the relevance/usefulness of the product or serx’ice) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2A. The paper or 

The paper or 


The paper or 


The paper or 


course focuses on 

course covers 


course covers 


course doesn’t 


topical areas 

high-priority 


mid-level- 


cover priority 


relevant to special 

topical areas 


priority topical 


topical areas 


education policy 

relevant to 


areas relevant 


relevant to 


or practice. 

special 


to special 


special 



education law, 


education law, 


education law, 



special 


special 


special 



education or 


education or 


education or 



education 


education 


education 



policy, 


policy, 


policy, 



including 


including 


including 



ESEA of 1965. 


ESEA of 1965. 


ESEA of 1965. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B. The paper or 

The paper or 


The paper or 


The paper or 


course uses 

course uses 


course uses 


course does 


national databases 

national 


national 


not use 


suitable for 

databases and 


databases and 


national 


analyses on 

data sets that 


data sets that 


databases and 


children with 

are well-suited 


are somewhat 


data sets that 


disabilities. 

for research on 


suitable for 


are appropriate 



children with 


research on 


for research on 



disabilities. 


children with 


children with 





disabilities. 


disabilities. 


Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2C.The product 

The 


The product 


The product 


or service 

information 


or service is 


or service 


contributes 

presented in 


somewhat 


does not 


new 

the product 


limited in 


provide 


information 

or service is 


providing new 


much if any 


to the 

new or 


information or 


new 


intended 

makes a 


in making a 


information 


audience on 

contribution 


contribution 


or does not 


the topic 

to the 


to the 


make a 


being 

intended 


intended 


contribution 


addressed. 

audience. 


audience. 


to the 

intended 

audience. 


Comments on rating, required for NA: 
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CONNECT 


INDICATORS AND SCORING RUBRIC FOR THE 
CENTER TO MOBILIZE EARLY CHILDHOOD KNOWLEDGE 

(CFDA 84.32 5 J) 


Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: PRODUCTS AND SERVICES DEEMED OF HIGH QUALITY: 
Synthesis rating (mean of 1A through 1H) 


Indicator 1A. The content of the product or service is based on rigorous and relevant 

evidence -based practices and on scientifically based research, when available. 


Indicator IB. The product or service uses an approach to technical assistance or professional 
development that is appropriate for promoting the desired change in policy or 
practice. 


Indicator 1C. The product or service reflects current theory and research on adult learning, 
including motivation, reinforcement, retention, transference, and use of 
reflective practice. 


Indicator ID. The product or service adequately addresses all of the research questions, 
hypotheses, or objectives presented. 


Indicator IE. The product or service’s findings are clearly supported by the data, or, for a 

concept or position paper or presentation, arguments or theories are backed up 
with valid research and sound reasoning. 


Indicator IF. The analyses conducted are appropriate for the data used in the product or 
service. 


Indicator 1G. The product or service’s data sources and instruments are appropriate for 
addressing the research questions, hypotheses, or objectives. 


Indicator 1H. The product or service is easy to read and/or understand, given the intended 
audience. 
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DIMENSION 2: PRODUCTS DEEMED OF HIGH RELEVANCE/USEFULNESS TO 
EDUCATIONAL OR EARLY INTERVENTION POLICY OR PRACTICE: Synthesis 
rating (mean of 2A through 2F) 


Indicator 2A. The product or service addresses a high-priority need. 


Indicator 2B. The product or service takes into account local conditions in which it will be 
used. 


Indicator 2C. The product or service can be easily integrated into existing resources for 
preservice preparation. 


Indicator 2D. The product or service is accessible to all targeted groups. 


Indicator 2E. The product or service includes a provision for technical assistance, for 
example, phone support or on-line help. 


Indicator 2F. The product or service contributes new information to the intended audience on 
the topic being addressed. 


GENERAL COMMENTS 


C-12 




CRITERIA FOR SCORING 


Dimension 1: Products and Services Deemed of High Quality ( Check the number that best describes 
the quality of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1A. The content 

In developing 


In developing the 


In developing 


of the 

the product or 


product or 


the product or 


product or 

service, the 


service, the 


service, the 


service is 

grantee 


grantee 


grantee rarely 


based on 

consistently 


sometimes used 


if ever used 


rigorous and 

used relevant 


relevant and 


relevant and 


relevant 

and rigorous 


rigorous research 


rigorous 


evidence- 

research with 


with appropriate 


research with 


based 

appropriate 


designs. 


appropriate 


practices and 

designs, 


methods, 


designs, 


on 

methods, 


measures, and 


methods, 


scientifically 

measures, and 


analyses over 


measures, or 


based 

analyses over 


those with 


analyses. 


research. 

research with 


weaker designs. 




when 

less relevance or 






available. 

weaker designs. 






Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IB. The product 

The product or 


The product or 


The product 


or service 

service uses an 


service uses an 


or service 


uses an 

approach to 


approach to 


uses an 


approach to 

technical 


technical 


approach to 


technical 

assistance or 


assistance or 


technical 


assistance or 

professional 


professional 


assistance or 


professional 

development 


development that 


professional 


development 

that is highly 


is moderately 


development 


that is 

appropriate for 


appropriate for 


that is not 


appropriate 

promoting the 


promoting the 


very 


for 

desired change 


desired change in 


appropriate 


promoting 

in policy or 


policy or 


for promoting 


the desired 

practice. 


practice, but 


the desired 


change in 



better 


change in 


policy or 



approaches could 


policy or 


practice. 



have been used. 


practice. 


Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1C. The product or 

The product or 


The product or 


The product 


service reflects 

service 


service 


or service 


current theory 

accurately 


accurately 


reflects few 


and research on 

reflects all 


reflects some 


if any 


adult learning, 

relevant 


but not all 


components 


including 

components of 


components of 


of adult 


motivation. 

adult learning 


adult learning 


learning 


reinforcement. 

theory. 


theory. 


theory. 


retention. 

including 


including 




transference, 

motivation. 


motivation. 




and use of 

reinforcement, 


reinforcement, 




reflective 

retention. 


retention. 




practice. 

transference of 


transference of 





content, and 


content, and 





facilitation of 


facilitation of 





reflective 


reflective 





practice. 


practice. 





Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

ID. The product or 

All of the 


Some but not 


Few if any 


service 

research 


all of the 


of the 


adequately 

questions, 


research 


research 


addresses all of 

hypotheses, or 


questions, 


questions. 


the research 

objectives are 


hypotheses, or 


hypotheses, 


questions. 

adequately 


objectives are 


or objectives 


hypotheses, or 

addressed. 


adequately 


are 


objectives 



addressed. 


adequately 


presented. 





addressed. 


Comments on rating, required for NA: 


C-15 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IE. The product or 

All of the 


Some but not 


Few if any of 


service’s 

findings are 


all of the 


the findings 


findings are 

clearly 


findings are 


are supported 


clearly 

supported by 


clearly 


by the data, 


supported by the 

the data, or, 


supported by 


or, for a 


data, or, for a 

for a concept 


the data, or, 


concept or 


concept or 

or position 


for a concept 


position 


position paper or 

paper, all 


or position 


paper, few if 


presentation. 

arguments or 


paper, some 


any of the 


arguments or 

theories are 


but not all of 


arguments or 


theories are 

backed up 


the 


theories are 


backed up with 

with valid 


arguments or 


backed up 


valid research 

research and 


theories are 


with valid 


and sound 

sound 


backed up 


research or 


reasoning. 

reasoning. 


with valid 


sound 





research and 


reasoning. 





sound 







reasoning. 




Comments on rating, required for NA: 


C-16 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IF. The analyses 

The product 


The analysis 


The product 


conducted are 

or service 


methods are 


or service 


appropriate for 

uses the best 


reasonable, 


uses few if 


the data used in 

appropriate 


although 


any 


the product or 

analysis 


better ones 


appropriate 


service. 

methods for 


could be 


methods to 



the research 


applied 


analyze the 



questions, 


within the 


data, or the 



hypotheses, 


context. 


analytic 



or objectives. 




methods are 







not described. 


Comments on rating, required for NA: 


C-17 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1G. The product 

All of the 


Some but not all 


The instruments 


or service’s 

instruments 


of the 


have little if any 


data sources 

are valid and 


instruments are 


validity or 


and 

reliable for 


valid and 


reliability for 


instruments 

addressing 


reliable; the 


addressing the 


are appropriate 

the research 


validity and/or 


research 


for addressing 

questions, 


reliability of 


questions, 


the research 

hypotheses, 


some 


hypotheses, or 


questions, 

or 


instruments is in 


objectives, and 


hypotheses, or 

objectives, 


question; or 


there are serious 


objectives. 

and the data 


there are some 


problems with 



are collected 


concerns with 


data collection, 



using sound 


the soundness of 


or 



practices. 


the data 


instrumentation 





collection. 


was applicable 
to the product or 
service but was 
not addressed at 

all. 



Comments on rating, required for NA: 


C-18 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

1H. The product 
or service is 
easy to read 
and 

understand, 
given the 
intended 
audience. 

The product 
or service is 
well written 
throughout 
and can be 
easily 
understood 
by the 
intended 
audience. 


Some but not all 
sections of the 
product or 
service are well 
written, and/or 
the information 
is too technical 
for the intended 
audience. 


The product or 
service is poorly 
written 
throughout. 



Comments on rating, required for NA: 


C-19 


Dimension 2: Products Deemed of High Relevance/Usefulness to Educational or Early Intervention 
Policy or Practice ( Check the number that best describes the relevance/usefulness of the product or 
sendee) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2A. The product 

The product 


The product 


The product 


or service 

or service 


or service 


or service 


addresses a 

addresses a 


addresses a 


does not 


high-priority 

critical need 


less critical 


address a 


need. 

of 


need of 


need of 



preservice 


preservice 


preservice 



trainees, 


trainees, 


trainees, 



practicing 


practicing 


practicing 



personnel, 


personnel, 


personnel, or 



or other 


or other 


other OSEP 



OSEP 


OSEP 


grantees. 



grantees. 


grantees, or 
it focuses 
indirectly on 
a critical 
need. 




Comments on rating, required for NA: 


C-20 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B. The product 

The product 


The product 


The product 


or service 

or service is 


or service is 


or service is 


takes into 

designed to 


designed to 


not designed 


account local 

achieve its 


achieve its 


in a way that 


conditions in 

professional 


professional 


will allow it 


which it will 

developmen 


development 


to achieve 


be used. 

t aims under 


aims under 


its 



most or all 


some local 


professional 



local 


conditions. 


development 



conditions. 




aims under 
most local 
conditions. 


Comments on rating, required for NA: 


C-21 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2C. The product 

The product or 


The product or 


The product or 


or service can 

service is easy 


service is 


service cannot 


be easily 

to integrate 


somewhat 


be integrated 


integrated 

into existing 


challenging to 


into existing 


into existing 

curricula and 


integrate into 


curricula and 


resources for 

experiential 


existing 


experiential 


professional 

opportunities, 


curricula and 


opportunities, 


development. 

professional 


experiential 


professional 



development 


opportunities, 


development for 



for practicing 


professional 


practicing 



personnel, a 


development 


personnel, a 



new product or 


for practicing 


new product or 



service, or 


personnel, a 


service, or 



existing 


new product or 


existing 



products or 


service, or 


products or 



services. 


existing 


services without 





products or 


substantial 





services. 


modifications. 


Comments on rating, required for NA: 


C-22 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2D. The product 

The product or 


Although it 


The product or 


or service is 

service is 


may be 


service is not 


accessible to 

readily 


available, the 


accessible to 


all targeted 

accessible to 


product or 


preservice 


groups when 

preservice 


service is not 


students, OSEP 


needed. 

students, 


easily 


grantees, and 



OSEP 


accessible to 


professionals in 



grantees, and 


preservice 


the field. 



professionals 


students, 





in the field, 


OSEP 





when needed. 


grantees, and 
professionals 
in the field, 
when needed. 





Comments on rating, required for NA: 


C-23 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2E.The product or 

Technical 


Technical 


Technical 


service 

assistance is 


assistance is 


assistance is 


includes a 

easily 


moderately 


very 


provision for 

accessible 


accessible to 


difficult to 


technical 

to potential 


potential 


access for 


assistance, for 

consumers, 


consumers or 


potential 


example, 

and is “user- 


is only 


consumers, 


phone support 

friendly.” 


moderately 


or the 


or on-line 



“user- 


technical 


help. 



friendly.” 


assistance is 

difficult to 
use. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2F.The paper or 

The 


The paper or 


The paper or 


presentation 

information 


presentation 


presentation 


contributes 

presented in 


is somewhat 


does not 


new 

the paper or 


limited in 


provide 


information to 

presentation 


providing 


much if any 


the intended 

is new or 


new 


new 


audience on 

makes a 


information 


information 


the topic being 

contribution 


or in making 


or does not 


addressed. 

to the 


a contribution 


make a 



intended 


to the 


contribution 



audience. 


intended 


to the 





audience. 


intended 

audience. 


Comments on rating, required for NA: 


C-24 


CTO 

INDICATORS AND SCORING RUBRIC FOR THE 
CENTER FOR IMPROVING TEACHER QUALITY 
(CFDA 84.325M) 


Product: 

Reviewer: Date: 


SUMMARY S H EET 


Dimensions and Indicators 

Rating 

DIMENSION 1: PRODUCTS AND SERVICES DEEMED OF HIGH QUALITY: 
Synthesis rating (mean of 1A through 1C) 


Indicator 1A. The content of the product or service is based on rigorous and relevant 

evidence-based practices and on scientifically based research, when available. 


Indicator IB. The product or service uses an approach to technical assistance or professional 
development that is appropriate for promoting the desired change in policy or 
practice. 


Indicator 1C. The product or service is aligned with a sound model for building statewide 
systems of training and improved licensure and certification. 



C-25 


DIMENSION 2: PRODUCTS DEEMED OF HIGH RELEVANCE/USEFULNESS TO 
EDUCATIONAL OR EARLY INTERVENTION POLICY OR PRACTICE: Synthesis 
rating (mean of 2A through 2E) 


Indicator 2A. The product or service addresses a high-priority need. 


Indicator 2B. The product or service is designed to engage or support a team of decision 
makers responsible for ensuring that regular and special educators are well 
prepared to promote learning for all students. 


Indicator 2C. The product or service is suitable for dissemination and/or use in other states or 
localities. 


Indicator 2D. The product or service can be reasonably expected to help improve preservice 
preparation, ongoing professional development, or accountability systems for 
teacher quality through changes in teacher preparation and/or certification and 
licensure. 


Indicator 2E. The product or service takes into account state environments in which it will be 
used. 


GENERAL COMMENTS 


C-26 




CRITERIA FOR SCORING 


Dimension 1: Products and Services Deemed of High Quality ( Check the number that best describes 
the quality of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1A. The content 

In developing 


In developing the 


In developing 


of the product 

the product or 


product or service, 


the product or 


or service is 

service, the 


the grantee 


service, the 


based on 

grantee 


sometimes used 


grantee rarely if 


rigorous and 

consistently used 


relevant and 


ever used 


relevant 

relevant and 


rigorous research 


relevant and 


evidence- 

rigorous 


with appropriate 


rigorous 


based 

research with 


designs, methods, 


research with 


practices and 

appropriate 


measures, and 


appropriate 


on 

designs, 


analyses over 


designs, 


scientifically 

methods, 


those with weaker 


methods. 


based 

measures, and 


designs. 


measures, or 


research. 

analyses over 




analyses. 


when 

research with 






available. 

less relevance or 
weaker designs. 






Comments on rating, required for NA: 


C-27 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IB. The product 

The product or 


The product or 


The product or 


or service 

service uses an 


service uses an 


service uses an 


uses an 

approach to 


approach to 


approach to 


approach to 

technical 


technical 


technical 


technical 

assistance or 


assistance or 


assistance or 


assistance or 

professional 


professional 


professional 


professional 

development 


development that 


development 


development 

that is highly 


is moderately 


that is not very 


that is 

appropriate for 


appropriate for 


appropriate for 


appropriate 

promoting the 


promoting the 


promoting the 


for promoting 

desired change 


desired change in 


desired change 


the desired 

in policy or 


policy or practice, 


in policy or 


change in 

practice. 


but better 


practice. 


policy or 



approaches could 




practice. 



have been used. 




Comments on rating, required for NA: 


C-28 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

lC.The product 

The product 


The product 


The product 


or service is 

or service is 


or service is 


or service is 


aligned with 

closely 


loosely 


unaligned 


a sound 

aligned with a 


aligned with 


with a sound 


model for 

sound model 


a sound 


model for 


building 

for building 


model for 


building 


statewide 

statewide 


building 


statewide 


systems of 

systems of 


statewide 


systems of 


training and 

training and 


systems of 


training and 


improved 

improved 


training and 


improved 


licensure and 

licensure and 


improved 


licensure and 


certification. 

certification. 


licensure and 


certification 





certification 


or is aligned 





or is closely 


with a model 





aligned with 


that is 





a model that 
is less sound. 


unsound. 



Comments on rating, required for NA: 


C-29 


Dimension 2: Products Deemed of High Relevance/Usefulness to Educational or Early Intervention 
Policy or Practice ( Check the number that best describes the relevance/usefulness of the product or 
sendee) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2A. The product 

The product 


The product 


The product 


or service 

or service 


addresses a 


or service 


addresses a 

addresses a 


less critical 


does not 


high-priority 

critical need 


need for 


address a 


need. 

for technical 


technical 


need for 



assistance to 


assistance to 


technical 



improve 


improve 


assistance to 



teacher 


teacher 


improve 



preparation 


preparation 


teacher 



and/or 


and/or 


preparation 



certification 


certification 


and/or 



and 


and 


certification 



licensure. 


licensure, or 


and 





it focuses 
indirectly on 
a critical 
need. 


licensure. 


Comments on rating, required for NA: 


C-30 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B. The product or 

The product or 


The product or 


It is likely that 


service is designed 

service is 


service may 


the product or 


to engage or 

clearly 


help a team of 


service will not 


support a team of 

designed for a 


decision-makers 


help a team of 


decision-makers 

team of 


responsible for 


decision- 


responsible for 

decision- 


ensuring that 


makers 


ensuring that 

makers 


regular and 


responsible for 


regular and special 

responsible for 


special 


ensuring that 


educators are well 

ensuring that 


educators are 


regular and 


prepared to 

regular and 


well prepared to 


special 


promote learning 

special 


promote 


educators are 


for all students. 

educators are 


learning for all 


well prepared 


Decision-makers 

well prepared 


students, but is 


to promote 


specified in the 

to promote 


not designed 


learning for all 


grant priority 

learning for all 


with that 


students. 


include elected 

students. 


specific purpose 




officials, faculty at 
teacher training 
institutions, 
personnel 
directors, and 
others. 



in mind. 




Comments on rating, required for NA: 


C-31 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2C. The product or 

The product or 


The product or 


The product or 


service is suitable 

service is well- 


service is 


service is not 


for dissemination 

suited for 


somewhat 


suitable for 


and/or use in other 

dissemination 


suited for 


dissemination 


states or localities. 

and/or use in 


dissemination 


or use in other 



other states or 


and/or use in 


states or 



localities. 


other states or 
localities. 


localities. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2D. The product or 

It is clear how 


It is somewhat 


It is 


service can be 

the product or 


unclear how 


completely 


reasonably 

service will 


the product or 


unclear how 


expected to help 

help improve 


service will 


the product or 


improve teacher 

teacher 


help improve 


service will 


preparation and/or 

preparation 


teacher 


help improve 


certification and 

and/or 


preparation 


teacher 


licensure either 

certification 


and/or 


preparation 


through preservice 

and licensure. 


certification 


and/or 


preparation. 



and licensure. 


certification 


ongoing 
professional 
development, or 
revised 
accountability 
systems for 
teacher quality. 





and licensure. 


Comments on rating, required for NA: 


C-32 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2E.The product or 

The product 


The product or 


The product 


service takes into 

or service 


service 


or service 


account state 

takes into 


recognizes the 


does not 


environments in 

account the 


variation in 


recognize or 


which it will be 

variation in 


state 


take into 


used. 

state 


environments 


account the 



environments 


under which it 


variation in 



under which 


will be used 


state 



it will be 


but does not 


environments 



used. 


take them into 


under which 





account fully. 


it will be 
used. 


Comments on rating, required for NA: 


C-33 


EI/ECSE 


INDICATORS AND SCORING RUBRIC FOR THE 

CENTER FOR EARLY CHILDHOOD EDUCATION/EARLY INTERVENTION PERSONNEL 

PREPARATION 

(CFDA 84.32 5 J) 


Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: CENTER QUALITY: Synthesis rating (mean of 1A through IF) 


Indicator 1A. The content of the product or service is based on rigorous and relevant evidence- 
based practices and on scientifically based research, when available. 


Indicator IB. The product or service adequately addresses all of the research questions, 
hypotheses, or objectives presented. 


Indicator 1C. The product or service’s findings are clearly supported by the data, or, for a concept 
or position paper or presentation, arguments or theories are backed up with valid 
research and sound reasoning. 


Indicator ID. The analyses conducted are appropriate for the data used in the paper or 
presentation. 


Indicator IE. The product or service’s data sources and instruments are appropriate for addressing 
the research questions, hypotheses, or objectives. 


Indicator IF. The product or service is easy to read and understand, given the intended audience. 





C-34 


DIMENSION 2: CENTER RELEVANCE/USEFULNESS: Synthesis rating (mean of 2A 
through 2D) 


Indicator 2A. The product or service addresses a high-priority need. 


Indicator 2B. The product or service enhances efforts to understand, develop, and/or implement 
EI/ECSE personnel development policies and practices. 


Indicator 2C. The product or service takes into account state and local conditions in which 
personnel preparation occurs. 


Indicator 2D. The product or service contributes new information to the intended audience on the 
topic being addressed. 


GENERAL COMMENTS 


C-35 




CRITERIA FOR SCORING 


Dimension 1: Center Quality ( Check the number that best describes the quality of the product or 
sendee ) 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

1A. The content 
of the 
product or 
service is 
based on 
rigorous and 
relevant 
evidence- 
based 

practices and 
on 

scientifically 

based 

research, 

when 

available. 

In developing 
the product 
or service, 
the grantee 
consistently 
used relevant 
and rigorous 
research with 
appropriate 
designs, 
methods, 
measures, 
and analyses 
over research 
with less 
relevance or 
weaker 
designs. 


In 

developing 
the product 
or service, 
the grantee 
sometimes 
used relevant 
and rigorous 
research with 
appropriate 
designs, 
methods, 
measures, 
and analyses 
over those 
with weaker 
designs. 


In developing 
the product 
or service, 
the grantee 
consistently 
did not use 
relevant and 
rigorous 
research with 
appropriate 
designs, 
methods, 
measures, or 
analyses. 



Comments on rating, required for NA: 


C-36 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IB. The product 
or service 
adequately 
addresses all 
of the 
research 
questions, 
hypotheses, 
or objectives 
presented. 

All of the 

research 

questions, 

hypotheses, 

or objectives 

are 

adequately 

addressed. 


Some but not 

all of the 

research 

questions, 

hypotheses, 

or objectives 

are 

adequately 

addressed. 


Few if any of 
the research 
questions, 
hypotheses, 
or objectives 
are 

adequately 

addressed. 


Comments on rating, required for NA: 






C-37 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

lC.The product 

All of the 


Some but not 


Few if any of 


or service’s 

findings are 


all of the 


the findings 


findings are 

clearly 


findings are 


are supported 


clearly 

supported by 


supported by 


by the data, or. 


supported by 

the data, or, for 


the data, or, 


for a for a 


the data, or. 

a concept or 


for a for a 


concept or 


for a concept 

position paper 


concept or 


position paper 


or position 

or presentation, 


position 


or 


paper or 

arguments or 


paper or 


presentation. 


presentation. 

theories are 


presentation, 


few if any of 


arguments or 

backed up with 


some but not 


the arguments 


theories are 

valid research 


all of the 


or theories are 


backed up 

and sound 


arguments or 


backed up 


with valid 

reasoning. 


theories are 


with valid 


research and 



backed up 


research and 


sound 



with valid 


sound 


reasoning. 



research and 

sound 

reasoning. 


reasoning. 



Comments on rating, required for NA: 


C-38 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

ID. The analyses 
conducted are 
appropriate 
for the data 
used in the 
product or 
service. 

The product or 
service uses the 
best appropriate 
analysis 
methods for the 
research 
questions, 
hypotheses, or 
objectives. 


The analysis 
methods are 
reasonable, 
although 
better ones 
could be 
applied 
within the 
context. 


The product or 
service uses 
few if any 
appropriate 
methods to 
analyze the 
data, or the 
analytic 
methods are 
not described. 


Comments on rating, required for NA: 






C-39 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IE. The product 

All of the 


Some but not 


The instruments 


or service’s 

instruments 


all of the 


have little if any 


data sources 

are valid and 


instruments 


validity or 


and 

reliable for 


are valid and 


reliability for 


instruments 

addressing 


reliable; the 


addressing the 


are 

the research 


validity 


research 


appropriate 

questions, 


and/or 


questions, 


for 

hypotheses, 


reliability of 


hypotheses, or 


addressing 

or objectives, 


some 


objectives, and 


the research 

and the data 


instruments is 


there are serious 


questions. 

are collected 


in question; 


problems with 


hypotheses, 

using sound 


or there are 


data collection, or 


or objectives. 

practices. 


some 


instrumentation 





concerns with 


was applicable to 





the soundness 


the product or 





of the data 


service but was 





collection. 


not addressed at 

all. 



Comments on rating, required for NA: 


C-40 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IF. The product 

The product 


Some but not 


The product or 


or service is 

or service is 


all sections of 


service is poorly 


easy to read 

well written 


the product 


written 


and 

throughout 


or service are 


throughout. 


understand, 

and can be 


well written. 




given the 

easily 


and/or the 




intended 

understood 


information 




audience. 

by the 


is too 





intended 


technical for 





audience. 


the intended 







audience. 




Comments on rating, required for NA: 


C-41 


Dimension 2: Center Relevance/Usefulness ( Check the number that best describes the 
relevance/usefulness of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2 A. The product 

The product or 


The product or 


The product or 


or service 

service 


service 


service does not 


addresses a 

addresses a 


addresses a less 


address a need 


high-priority 

critical need 


critical need 


with regard to 


need. 

with regard to 


with regard to 


EI/ECSE 



EI/ECSE 


EI/ECSE 


personnel 



personnel 


personnel 


development. 



development. 


development, 
or it focuses 
indirectly on a 
critical need. 




Comments on rating, required for NA: 


C-42 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B. The product 

The product or 


The product or 


The product or 


or service 

service directly 


service 


service does not 


enhances 

enhances 


indirectly 


enhance 


efforts to 

understanding. 


enhances 


understanding, 


understand, 

development. 


understanding. 


development. 


develop, 

and/or 


development, 


and/or 


and/or 

implementation 


and/or 


implementation 


implement 

of EI/ECSE 


implementation 


of EI/ECSE 


EI/ECSE 

personnel 


of EI/ECSE 


personnel 


personnel 

development 


personnel 


development 


development 

policies and/or 


development 


policies or 


policies and 

practices. 


policies and/or 


practices. 


practices. 



practices. 




Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2C.The product or 

The product 


The product 


The product 


service takes 

or service 


or service 


or service 


into account 

takes into 


recognizes the 


does not 


state and local 

account the 


state and local 


recognize or 


conditions in 

state and 


conditions 


take into 


which 

local 


under which 


account the 


personnel 

conditions 


personnel 


state or local 


preparation 

under which 


preparation 


conditions 


occurs. 

personnel 


occurs but 


under which 



preparation 


does not take 


personnel 



occurs. 


them into 


preparation 





account fully. 


occurs. 


Comments on rating, required for NA: 


C-43 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2D. The product 

The 


The product 


The product 


or service 

information 


or service is 


or service 


contributes 

presented in 


somewhat 


does not 


new 

the product 


limited in 


provide any 


information to 

or service is 


providing new 


new 


the intended 

new or 


information or 


information 


audience on 

makes a 


in making a 


or does not 


the topic being 

contribution 


contribution to 


make a 


addressed. 

to the 


the intended 


contribution 



intended 


audience. 


to the 



audience. 




intended 

audience. 


Comments on rating, required for NA: 


C-44 


IR1S-I 


INDICATORS AND SCORING RUBRIC FOR THE 

IDEA AND RESEARCH FOR INCLUSIVE SETTINGS CENTER FOR FACULTY 

ENHANCEMENT 

(CFDA 84.325F) 


Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: PRODUCTS AND SERVICES DEEMED OF HIGH QUALITY: 
Synthesis rating (mean of 1A through ID) 


Indicator 1A. The content of the product or service is based on rigorous and relevant 

evidence-based practices and on scientifically based research, when available. 


Indicator IB. The product or service uses an approach to technical assistance or professional 
development that is appropriate for promoting the desired change in policy or 
practice. 


Indicator 1C. The product or service reflects current theory and research on adult learning, 
including motivation, reinforcement, retention, transference, and use of 
reflective practice. 


Indicator ID. The product or service uses promising new strategies, or builds off existing 
ones, for supporting preservice preparation. 




DIMENSION 2: PRODUCTS DEEMED OF HIGH RELEVANCE/USEFULNESS TO 
POLICY OR PRACTICE: Synthesis rating (mean of 2A through 2F) 


Indicator 2A. The product or service addresses a high priority need for enhanced professional 
development. 


Indicator 2B. The product or service can be easily integrated into existing resources for 
preservice preparation. 


Indicator 2C. The product or service is accessible to all targeted groups. 



C-45 


Indicator 2D. The product or service incorporates state-of-the-art technology to enhance 
distribution. 


Indicator 2E. The product or service takes into account local conditions in which it will be 
used. 


Indicator 2F. The product or service includes a provision for technical assistance, for 
example, phone support or on-line help. 




GENERAL COMMENTS 


C-46 




CRITERIA FOR SCORING 


Dimension 1: Products and Services Deemed of High Quality ( Check the number that best describes 
the quality of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1A. The content of 

In developing 


In developing the 


In developing 


the product or 

the product or 


product or service, 


the product or 


service is based 

service, the 


the grantee 


service, the 


on rigorous and 

grantee 


sometimes used 


grantee rarely if 


relevant 

consistently 


relevant and 


ever used 


evidence-based 

used relevant 


rigorous research 


relevant and 


practices and on 

and rigorous 


with appropriate 


rigorous 


scientifically 

research with 


designs, methods, 


research with 


based research, 

appropriate 


measures, and 


appropriate 


when available. 

designs, 


analyses over 


designs, 



methods, 


those with weaker 


methods. 



measures, and 


designs. 


measures, or 



analyses over 
research with 
less relevance 
or weaker 
designs. 




analyses. 



Comments on rating, required for NA: 


C-47 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IB. The product or 

The product or 


The product or 


The product or 


service uses an 

service uses an 


service uses an 


service uses an 


approach to 

approach to 


approach to 


approach to 


technical 

technical 


technical 


technical 


assistance or 

assistance or 


assistance or 


assistance or 


professional 

professional 


professional 


professional 


development 

development 


development that 


development 


that is 

that is highly 


is moderately 


that is not very 


appropriate for 

appropriate for 


appropriate for 


appropriate for 


promoting the 

promoting the 


promoting the 


promoting the 


desired change 

desired change 


desired change in 


desired change 


in policy or 

in policy or 


policy or practice, 


in policy or 


practice. 

practice. 


but better 


practice. 





approaches could 







have been used. 





Comments on rating, required for NA: 


C-48 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1C. The product or 

The product or 


The product or 


The product 


service reflects 

service 


service 


or service 


current theory 

accurately 


accurately 


reflects few if 


and research on 

reflects all 


reflects some but 


any 


adult learning, 

relevant 


not all 


components 


including 

components of 


components of 


of adult 


motivation. 

adult learning 


adult learning 


learning 


reinforcement. 

theory, including 


theory, including 


theory. 


retention. 

motivation. 


motivation. 




transference, 

reinforcement. 


reinforcement. 




and use of 

retention. 


retention. 




reflective 

transference of 


transference of 




practice. 

content, and 


content, and 





facilitation of 


facilitation of 





reflective 


reflective 





practice. 


practice. 




Comments on rating, required for NA: 


C-49 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable/ 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

ID. The product or 
service uses 
promising new 
strategies, or 
builds off 
existing ones, 
for supporting 
preservice 
preparation. 

It is clear how 
the product or 
service is either a 
promising new 
strategy for 
supporting 
preservice 
preparation or 
one that builds 
substantially 
upon previous 
strategies. 


The product or 
service is not a 
new strategy for 
supporting 
preservice 
preparation but it 
builds slightly 
upon previous 
strategies. 


The product 
or service 
simply 
replicates an 
old strategy 
for supporting 
preservice 
preparation. It 
is not a 
promising 
new strategy 
nor does it 
build upon 
previous 
strategies. 



Comments on rating, required for NA: 


C-50 


Dimension 2: Products Deemed of High Relevance/Usefulness to Policy or Practice ( Check the 
number that best describes the relevance/usefulness of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2A. The product or 

The product or 


The product or 


The product or 


service 

service 


service addresses 


service does 


addresses a 

addresses a 


a less critical 


not address a 


high-priority 

critical need of 


need of IHE 


need of IHE 


need for 

IHE faculty 


faculty related to 


faculty related 


enhanced 

related to their 


their efforts to 


to their efforts 


professional 

efforts to 


provide training 


to provide 


development. 

provide training 


to school 


training to 



to school 


administrators, 


school 



administrators, 


regular education 


administrators, 



regular 


teachers, school 


regular 



education 


counselors, or 


education 



teachers, school 


school nurses; or 


teachers, 



counselors, or 


it focuses 


school 



school nurses. 


indirectly on a 


counselors, or 





critical need. 


school nurses. 


Comments on rating, required for NA: 


C-51 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B.The product or 

The product 


The product 


The product 


service can be 

or service 


or service is 


or service 


easily 

can be easily 


somewhat 


cannot be 


integrated into 

integrated 


challenging 


integrated 


existing 

into existing 


to integrate 


into existing 


resources for 

curricula and 


into existing 


curricula and 


preservice 

experiential 


curricula and 


experiential 


preparation. 

opportunities 


experiential 


opportunities 



with only 


opportunities 


without 



slight 


and will 


substantial 



modification 


require more 


modifications 



s to the new 


than slight 


to the new 



product or 


modifications 


product or 



service or to 


to the new 


service or to 



existing 


product or 


existing ones. 



ones. 


service or to 







existing ones. 




Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2C.The product 

The product 


Although it 


Few if any 


or service is 

or service is 


may be 


IHE faculty 


accessible to 

easy for 


available, the 


can obtain 


all targeted 

IHE faculty 


product or 


or use the 


groups. 

to obtain 


service is not 


product or 



and use, 


easy for IHE 


service 



when 


faculty to 


when 



needed. 


obtain or 
use, when 
needed. 


needed. 


Comments on rating, required for NA: 


C-52 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2D. The product 

The product 


The product 


The product 


or service 

or service 


or service 


or service 


incorporates 

fully 


incorporates 


does not 


state-of-the- 

incorporates 


some 


incorporate 


art technology 

state-of-the- 


technology 


technology 


to enhance 

art 


to enhance 


in a way that 


dissemination. 

technology 


disseminatio 


substantially 



to enhance 


n and use. 


enhances 



disseminati 




disseminatio 



on and use. 




n or use. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 1 . 

2E.The product 

The product 


The product 


The product 


or service 

or service 


or service 


or service 


takes into 

takes into 


recognizes 


does not 


account local 

account the 


the variation 


recognize or 


conditions in 

variation in 


in local 


take into 


which it will 

local 


conditions 


account the 


be used. 

conditions 


under which 


variation in 



under which 


it will be 


local 



it will be 


used but 


conditions 



used. 


does not 


under which 





take them 


it will be 





into account 


used. 





fully. 




Comments on rating, required for NA: 


C-53 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2F. The product 

Technical 


Technical 


Technical 


or service 

assistance is 


assistance is 


assistance is 


includes a 

easily 


moderately 


very 


provision for 

accessible 


accessible to 


difficult to 


technical 

to potential 


potential 


access for 


assistance, for 

consumers, 


consumers 


potential 


example, 

and is “user- 


or is only 


consumers 


phone support 

friendly.” 


moderately 


or the 


or on-line 



“user- 


technical 


help. 



friendly.” 


assistance is 
difficult to 
use. 


Comments on rating, required for NA: 


C-54 


IR1S-II 


INDICATORS AND SCORING RUBRIC FOR THE 

IDEA AND RESEARCH FOR INCLUSIVE SETTINGS CENTER FOR TRAINING 

ENHANCEMENTS 

(CFDA 84.325F) 


Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: PRODUCTS AND SERVICES DEEMED OF HIGH QUALITY: 
Synthesis rating (mean of 1A through ID) 


Indicator 1A. The content of the product or service is based on rigorous and relevant 

evidence-based practices and on scientifically based research, when available. 


Indicator IB. The product or service uses an approach to technical assistance or professional 
development that is appropriate for promoting the desired change in policy or 
practice. 


Indicator 1C. The product or service reflects current theory and research on adult learning, 
including motivation, reinforcement, retention, transference, and use of 
reflective practice. 


Indicator ID. The product or service facilitates implementation of programs to close 

achievement gaps for students with disabilities and to promote their education in 
the least restrictive environment. 




DIMENSION 2: PRODUCTS DEEMED OF HIGH RELEVANCE/USEFULNESS TO 
POLICY OR PRACTICE: Synthesis rating (mean of 2A through 2E) 


Indicator 2A. The product or service addresses a high-priority need. 


Indicator 2B. The product or service can be easily integrated into existing resources for 
preservice preparation and professional development. 


Indicator 2C. The product or service is accessible to all targeted groups. 



C-55 




CRITERIA FOR SCORING 


Dimension 1: Products and Services Deemed of High Quality ( Check the number that best describes 
the quality of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 1 

□ 

□ 

□ 

□ 

□ 

1A. The content of 

in developing the 


In developing the 


In developing 


the product or 

product or 


product or 


the product or 


service is based 

service, the 


service, the 


service, the 


on rigorous and 

grantee 


grantee 


grantee rarely 


relevant 

consistently used 


sometimes used 


if ever used 


evidence-based 

relevant and 


relevant and 


relevant and 


practices and on 

rigorous research 


rigorous research 


rigorous 


scientifically 

with appropriate 


with appropriate 


research with 


based research. 

designs, 


designs, methods, 


appropriate 


when available. 

methods, 


measures, and 


designs, 



measures, and 


analyses over 


methods. 



analyses over 


those with weaker 


measures, or 



research with 
less relevance or 
weaker designs. 


designs. 


analyses. 


Comments on rating, required for NA: 


C-57 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ | 

□ 

□ 

□ 

□ 

IB. The product 

The product or 


The product or 


The product or 


or service 

service uses an 


service uses an 


service uses 


uses an 

approach to 


approach to 


an approach to 


approach to 

technical 


technical 


technical 


technical 

assistance or 


assistance or 


assistance or 


assistance or 

professional 


professional 


professional 


professional 

development that 


development that 


development 


development 

is highly 


is moderately 


that is not 


that is 

appropriate for 


appropriate for 


very 


appropriate 

promoting the 


promoting the 


appropriate 


for promoting 

desired change 


desired change in 


for promoting 


the desired 

in policy or 


policy or practice, 


the desired 


change in 

practice. 


but better 


change in 


policy or 



approaches could 


policy or 


practice. 



have been used. 


practice. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1C. The product or 

The product or 


The product or 


The product or 


service reflects 

service 


service 


service reflects 


current theory 

accurately 


accurately 


few if any 


and research on 

reflects all 


reflects some but 


components of 


adult learning, 

relevant 


not all 


adult learning 


including 

components of 


components of 


theory. 


motivation. 

adult learning 


adult learning 




reinforcement. 

theory, including 


theory, including 




retention. 

motivation. 


motivation. 




transference, 

reinforcement, 


reinforcement. 




and use of 

retention, 


retention. 




reflective 

transference of 


transference of 




practice. 

content, and use 


content, and use 





of reflective 


of reflective 





practice. 


practice. 





Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

ID. The product or 

The product or 


The product or 


The product or 


service 

service helps 


service helps 


service neither 


facilitates 

both prepare 


either prepare 


prepares highly 


implementation 

highly qualified 


highly qualified 


qualified teachers 


of programs to 

teachers so that 


teachers so that 


so that LEAs 


close 

LEAs have the 


LEAs have the 


have the capacity 


achievement 

capacity to 


capacity to 


to implement 


gaps for 

implement school 


implement school 


school 


students with 

improvement 


improvement 


improvement 


disabilities and 

programs to close 


programs to close 


programs to close 


to promote their 

achievement gaps 


achievement gaps 


achievement gaps 


education in the 

and goals of 


and goals of 


and goals of 


least restrictive 

NCLB AND 


NCLB OR 


NCLB nor 


environment. 

promote access to 


promote access to 


promotes access 



and greater 


and greater 


to and greater 



participation and 


participation and 


participation and 



progress in the 


progress in the 


progress in the 



general education 


general education 


general education 



curriculum in the 


curriculum in the 


curriculum in the 



least restrictive 


least restrictive 


least restrictive 



environment. 


environment. 


environment. 


Comments on rating, required for NA: 


C-59 


Dimension 2: Products Deemed of High Relevance/Usefulness to Policy or Practice ( Check the 
number that best describes the relevance/usefulness of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2A. The product 

The product 


The product 


The product 


or service 

or service 


addresses a 


or service 


addresses a 

addresses a 


less critical 


does not 


high-priority 

critical need 


need ofIHE 


address a 


need. 

ofIHE 


faculty and 


need of IHE 



faculty and 


professional 


faculty or 



professional 


development 


professional 



development 


providers, or 


development 



providers. 


it focuses 
indirectly on a 
critical need. 


providers. 


Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B.The product or 

The product 


The product 


The product 


service can be 

or service 


or service is 


or service 


easily 

can be easily 


somewhat 


cannot be 


integrated into 

integrated 


challenging to 


integrated 


existing 

into existing 


integrate into 


into existing 


resources for 

curricula and 


existing 


curricula and 


preservice 

experiential 


curricula and 


experiential 


preparation 

opportunities 


experiential 


opportunities 


and 

with only 


opportunities 


without 


professional 

slight 


and will 


substantial 


development. 

modification 


require more 


modifications 



s to the new 


than slight 


to the new 



product or 


modifications 


product or 



service or to 


to the new 


services or to 



existing 


product or 


existing ones. 



ones. 


service or to 







existing ones. 




Comments on rating, required for NA: 


C-61 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2C.The product or 

The product 


Although it 


Few if any 


service is 

or service is 


may be 


IHE faculty 


accessible to 

easy for 


available, 


can obtain or 


all targeted 

IHE faculty 


the product 


use the 


groups. 

to obtain 


or service is 


product or 



and use, 


not easy for 


service when 



when 


IHE faculty 


needed. 



needed. 


to obtain or 
use, when 
needed. 




Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2D. The product 

Technical 


Technical 


Technical 


or service 

assistance is 


assistance is 


assistance is 


includes a 

easily 


moderately 


very difficult 


provision for 

accessible to 


accessible to 


to access for 


technical 

potential 


potential 


potential 


assistance. 

consumers 


consumers 


consumers, 



and is “user- 


or is only 


or the 



friendly.” 


moderately 


technical 





“user- 


assistance is 





friendly.” 


difficult to 







use. 


Comments on rating, required for NA: 


C-62 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2E.The product or 

The product 


The product 


The product 


service takes 

or service 


or service 


or service 


into account 

takes into 


recognizes 


does not 


local 

account the 


the local 


recognize or 


conditions in 

local 


conditions 


take into 


which it will 

conditions 


under which 


account the 


be used. 

under which 


it will be used 


local 



it will be 


but does not 


conditions 



used. 


take them 


under which 





into account 


it will be 





fully. 


used. 


Comments on rating, required for NA: 


C-63 


ITALD 


INDICATORS AND SCORING RUBRIC FOR THE 
INTERDISCIPLINARY TRAINING IN ANALYSIS OF LARGE-SCALE DATABASES 

(CFDA 84.325L) 


Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: CENTER QUALITY: Synthesis rating (mean of 1A through 1G) 


Indicator 1A. The course or courses, enrichment activities, and/or experiential learning 
opportunities cover key content areas relevant to conducting large-scale 
analyses in special education. 


Indicator IB. The faculty members or other individuals delivering course content were 
qualified for their assigned role. 


Indicator 1C. The product or service adequately addresses all of the research questions, 
hypotheses, or objectives presented. 


Indicator ID. The product or service’s findings are clearly supported by the data. 


Indicator IE. The analyses conducted are appropriate for the data used in the product or 
service. 


Indicator IF. The product or service’s data sources and instruments are appropriate for 
addressing the research questions, hypotheses, or objectives. 


Indicator 1G. The product or service is easy to read and/or understand, given the intended 
audience. 
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DIMENSION 2: CENTER RELEVANCE/USEFULNESS: Synthesis rating (mean of 2A 
through 2C) 


Indicator 2A. The paper or course focuses on topical areas relevant to special education policy 
or practice. 


Indicator 2B. The paper or course uses national databases suitable for analyses on children 
with disabilities. 


Indicator 2C. The product or service contributes new information to the intended audience on 
the topic being addressed. 




GENERAL COMMENTS 


C-65 




CRITERIA FOR SCORING 


Dimension 1: Center Quality, ( Check the number that best describes the quality of the product or 
sendee) 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 1 

□ 

□ 

□ 

□ 

□ 

1A. The course or 

The course or 


The course or 


The course or 


courses, 

courses, 


courses, 


courses, 


enrichment 

enrichment 


enrichment 


enrichment 


activities, 

activities, 


activities, and/or 


activities, 


and/or 

and/or 


experiential 


and/or 


experiential 

experiential 


learning 


experiential 


learning 

learning 


opportunities 


learning 


opportunities 

opportunities 


cover some 


opportunities 


cover key 

cover key 


content relevant to 


cover little if 


content relevant 

content 


conducting large- 


any content 


to conducting 

relevant to 


scale analyses in 


relevant to 


large-scale 

conducting 


special education 


conducting 


analyses in 

large-scale 


but omit important 


large-scale 


special 

analyses in 


content or include 


analyses in 


education. 

special 


unnecessary 


special 



education. 


content. 


education. 


Comments on rating, required for NA: 
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Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

IB. The faculty 
members or 
other 

individuals 
delivering 
course content 
were qualified 
for their 
assigned role. 

The faculty 
members or 
other 

individuals 
delivering 
course content 
were highly 
qualified for 
their assigned 
role. 


The faculty 
members or other 
individuals 
delivering course 
content were 
moderately 
qualified for their 
assigned role. 


The faculty 
members or 
other 

individuals 
delivering 
course content 
were poorly 
qualified for 
their assigned 
role. 



Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

lC.The product 
or service 
adequately 
addresses all 
of the 
research 
questions, 
hypotheses, 
or objectives 
presented. 

All of the 

research 

questions, 

hypotheses, 

or objectives 

are 

adequately 

addressed. 


Some but not 

all of the 

research 

questions, 

hypotheses, 

or objectives 

are 

adequately 

addressed. 


Few if any of 
the research 
questions, 
hypotheses, 
or objectives 
are 

adequately 

addressed. 


Comments on rating, required for NA: 
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Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

ID. The product 
or service’s 
findings are 
clearly 
supported by 
the data. 

All of the 
findings are 
clearly 
supported by 
the data. 


Some but not 
all of the 
findings are 
supported by 
the data. 


Few if any of 
the findings 
are supported 
by the data. 


Comments on rating, required for NA: 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

IE. The analyses 
conducted are 
appropriate for 
the data used in 
the. product or 
service. 

The product 
or service 
uses the best 
and 

appropriate 

analysis 

methods for 

the research 

questions, 

hypotheses, 

or 

objectives. 


The analysis 
methods are 
reasonable, 
although better 
ones could be 
applied within 
the context. 


The product or 
service uses few 
if any 

appropriate 
methods to 
analyze the data 
or the analytic 
methods are not 
described. 



Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IF. The product or 

All of the 


Some but not 


The instruments 


service’s data 

instruments 


all of the 


have little if any 


sources and 

are valid and 


instruments 


validity or 


instruments are 

reliable for 


are valid and 


reliability for 


appropriate for 

addressing 


reliable; the 


addressing the 


addressing the 

the research 


validity and/or 


research 


research 

questions, 


reliability of 


questions, 


questions. 

hypotheses, 


some 


hypotheses, or 


hypotheses, or 

or 


instruments is 


objectives, and 


objectives. 

objectives, 


in question; or 


there are serious 



and the data 


there are some 


problems with 



are collected 


concerns with 


data collection, 



using sound 


the soundness 


or 



practices. 


of the data 


instrumentation 





collection. 


was applicable to 
the paper or 
presentation but 
was not 

addressed at all. 



Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ l 

□ 

□ 

□ 

1G. The product 

The product 


Some but not 


The product 


or service is 

or service is 


all sections of 


or service is 


easy to read 

well written 


the product or 


poorly 


and 

throughout 


service are 


written 


understand, 

and can be 


well written, 


throughout. 


given the 

easily 


and/or the 




intended 

understood 


information is 




audience. 

by the 


too technical 





intended 


for the 





audience. 


intended 







audience. 




Comments on rating, required for NA: 


C-70 


Dimension 2: Center Relevance/Usefulness ( Check the number that best describes the 
relevance/usefulness of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2A. The paper or 

The paper 


The paper or 


The paper or 


course focuses 

or course 


course 


course 


on topical 

covers high- 


covers mid- 


doesn’t 


areas relevant 

priority 


level- 


cover 


to special 

topical areas 


priority 


priority 


education 

relevant to 


topical areas 


topical areas 


policy or 

special 


relevant to 


relevant to 


practice. 

education 


special 


special 



law, special 


education 


education 



education. 


law, special 


law, special 



or education 


education. 


education, or 



policy, 


or education 


education 



including 
ESEA of 
1965 and 
NCLB. 


policy. 


policy. 


Comments on rating, required for NA: 


C-71 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B.The paper or 

The paper 


The paper or 


The paper or 


course uses 

or course 


course uses 


course does 


national 

uses 


national 


not use 


databases 

national 


databases 


national 


suitable for 

databases 


and data sets 


databases 


analyses on 

and data 


that are 


and data sets 


children with 

sets that are 


somewhat 


that are 


disabilities. 

well-suited 


suitable for 


appropriate 



for research 


research on 


for research 



on children 


children 


on children 



with 


with 


with 



disabilities. 


disabilities. 


disabilities. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2C.The product or 

The 


The paper or 


The paper or 


service 

information 


presentation is 


presentation 


contributes 

presented in 


somewhat 


does not 


new 

the paper or 


limited in 


provide 


information to 

presentation 


providing new 


much if any 


the intended 

is new or 


information or 


new 


audience on 

makes a 


in making a 


information 


the topic being 

contribution 


contribution to 


or does not 


addressed. 

to the 


the intended 


make a 



intended 


audience. 


contribution 



audience. 




to the 

intended 

audience. 


Comments on rating, required for NA: 


C-72 


NCIPP 


INDICATORS AND SCORING RUBRIC FOR THE 

NATIONAL CENTER TO INFORM POLICY AND PRACTICE IN SPECIAL EDUCATION 

PROFESSIONAL DEVELOPMENT 

(CFDA 84.325Q) 


Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: PRODUCTS AND SERVICES DEEMED OF HIGH QUALITY: 
Synthesis rating (mean of 1A through IF) 


Indicator 1A. The content of the product or service is based on rigorous and relevant 

evidence-based practices and on scientifically based research, when available. 


Indicator IB. The product or service adequately addresses all of the research questions, 
hypotheses, or objectives presented. 


Indicator 1C. The product or service’s findings are clearly supported by the data or, for a 

concept or position paper or presentation, arguments or theories are backed up 
with valid research and sound reasoning. 


Indicator ID. The analyses conducted are appropriate for the data used in the product or 
service. 


Indicator IE. The product or service’s data sources and instruments are appropriate for 
addressing the research questions, hypotheses, or objectives. 


Indicator IF. The product or service is easy to read and/or understand, given the intended 
audience. 




DIMENSION 2: PRODUCTS DEEMED OF HIGH RELEVANCE/USEFULNESS TO 
POLICY OR PRACTICE: Synthesis rating (mean of 2A through 2D) 


Indicator 2A. The product or service addresses a high-priority need. 


Indicator 2B. The product or service enhances efforts to understand, develop, and/or 
implement induction and mentoring policies and practices. 
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Indicator 2C. The product or service takes into account local conditions in which induction 
and mentoring occur. 


Indicator 2D. The product or service contributes new information to the intended audience on 
the topic being addressed. 




GENERAL COMMENTS 


C-74 




CRITERIA FOR SCORING 


Dimension 1: Products and Services Deemed of High Quality ( Check the number that best describes 
the quality of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1A. The content 

In developing 


In developing 


In developing 


of the product 

the product or 


the product or 


the product or 


or service is 

service, the 


service, the 


service, the 


based on 

grantee 


grantee 


grantee rarely 


rigorous and 

consistently 


sometimes used 


if ever used 


relevant 

used relevant 


relevant and 


relevant and 


evidence- 

and rigorous 


rigorous 


rigorous 


based 

research with 


research with 


research with 


practices and 

appropriate 


appropriate 


appropriate 


on 

designs, 


designs, 


designs. 


scientifically 

methods, 


methods, 


methods, 


based 

measures, and 


measures, and 


measures, or 


research. 

analyses over 


analyses over 


analyses. 


when 

research with 


those with 




available. 

less relevance 
or weaker 
designs. 


weaker designs. 





Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IB. The product 
or service 
adequately 
addresses all 
of the 
research 
questions, 
hypotheses, 
or objectives 
presented. 

All of the 

research 

questions, 

hypotheses, 

or objectives 

are 

adequately 

addressed. 


Some but not 
all of the 
research 
questions, 
hypotheses, or 
objectives are 
adequately 
addressed. 


Few if any of 
the research 
questions, 
hypotheses, 
or objectives 
are 

adequately 

addressed. 


Comments on rating, required for NA: 







Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1C. The product or 

All of the 


Some but not all 


Few if any of 


service’s 

findings are 


of the findings 


the findings 


findings are 

clearly 


are clearly 


are supported 


clearly 

supported by 


supported by 


by the data, or. 


supported by 

the data, or, 


the data, or, for 


for a concept 


the data, or, 

for a concept 


a concept or 


or position 


for a concept 

or position 


position paper, 


paper, few if 


or position 

paper, all 


some but not all 


any of the 


paper, 

arguments or 


of the 


arguments or 


arguments or 

theories are 


arguments or 


theories are 


theories are 

backed up 


theories are 


backed up with 


backed up with 

with valid 


backed up with 


valid research 


valid research 

research and 


valid research 


or sound 


and sound 

sound 


and sound 


reasoning. 


reasoning. 

reasoning. 


reasoning. 




Comments on rating, required for NA: 


C-76 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

ID. The analyses 
conducted are 
appropriate for 
the data used 
in the product 
or service. 

The product 
or service 
uses the best 
appropriate 
analysis 
methods for 
the research 
questions, 
hypotheses, 
or objectives. 


The analysis 
methods are 
reasonable, 
although better 
ones could be 
applied within 
the context. 


The product or 
service uses 
few if any 
appropriate 
methods to 
analyze the 
data, or the 
analytic 
methods are 
not described. 


Comments on rating, required for NA: 







Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IE. The product 

All of the 


Some but not 


The instruments 


or service’s 

instruments 


all of the 


have little if any 


data sources 

are valid and 


instruments 


validity or 


and 

reliable for 


are valid and 


reliability for 


instruments 

addressing 


reliable; the 


addressing the 


are 

the research 


validity and/or 


research 


appropriate 

questions, 


reliability of 


questions, 


for 

hypotheses, 


some 


hypotheses, or 


addressing 

or objectives, 


instruments is 


objectives, and 


the research 

and the data 


in question; or 


there are serious 


questions. 

are collected 


there are some 


problems with 


hypotheses, 

using sound 


concerns with 


data collection, or 


or objectives. 

practices. 


the soundness 


instrumentation 





of the data 


was applicable to 





collection. 


the product or 
service but was 
not addressed at 

all. 



Comments on rating, required for NA: 
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Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

IF. The product 
or service is 
easy to read 
and 

understand, 
given the 
intended 
audience. 

The product 
or service is 
well written 
throughout 
and can be 
easily 
understood 
by the 
intended 
audience. 


Some but not 
all sections of 
the product or 
service are 
well written, 
and/or the 
information is 
too technical 
for the 
intended 
audience. 


The product or 
service is poorly 
written 
throughout. 



Comments on rating, required for NA: 


C-78 


Dimension 2: Products Deemed of High Relevance to Policy or Practice ( Check the number that best 
describes the relevance/usefulness of the product or service) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2 A. The 

The product or 


The product or 


The product or 


product or 

service 


service 


service does not 


service 

addresses a 


addresses a less 


address a need 


addresses a 

critical need 


critical need 


with regard to 


high- 

with regard to 


with regard to 


induction and 


priority 

induction and 


induction and 


mentoring. 


need. 

mentoring. 


mentoring, or it 
focuses 
indirectly on a 
critical need. 




Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B.The product 

The product or 


The product or 


The product or 


or service 

service directly 


service 


service does not 


enhances 

enhances 


indirectly 


enhance 


efforts to 

understanding. 


enhances 


understanding, 


understand, 

development. 


understanding, 


development. 


develop, 

and/or 


development. 


and/or 


and/or 

implementation 


and/or 


implementation 


implement 

of induction or 


implementation 


of induction or 


induction 

mentoring 


of induction or 


mentoring 


and 

policies and/or 


mentoring 


policies or 


mentoring 

practices. 


policies and/or 


practices. 


policies and 
practices. 



practices. 




Comments on rating, required for NA: 


C-79 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2C.The product 

The product 


The product or 


The product 


or service 

or service 


service 


or service 


takes into 

takes into 


recognizes the 


does not 


account 

account the 


variation in 


recognize or 


local 

variation in 


local 


take into 


conditions 

local 


conditions 


account the 


in which 

conditions 


under which 


variation in 


induction 

under which 


induction and 


local 


and 

induction 


mentoring 


conditions 


mentoring 

and 


occur but does 


under which 


occur. 

mentoring 


not take them 


induction and 



occur. 


into account 


mentoring 





fully. 


occur. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2D. The product 

The 


The product or 


The product 


or service 

information 


service is 


or service 


contributes 

presented in 


somewhat 


does not 


new 

the product 


limited in 


provide much 


information 

or service is 


providing new 


if any new 


to the 

new or 


information or 


information 


intended 

makes a 


in making a 


or does not 


audience on 

contribution 


contribution to 


make a 


the topic 

to the 


the intended 


contribution 


being 

intended 


audience. 


to the 


addressed. 

audience. 




intended 

audience. 


Comments on rating, required for NA: 


C-80 


NCLVI 


INDICATORS AND SCORING RUBRIC FOR THE 

NATIONAL CENTER FOR LEADERSHIP IN VISUAL IMPAIRMENT 

(CFDA 84.325U) 


Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: PRODUCTS AND SERVICES DEEMED OF HIGH QUALITY: 
Synthesis rating (mean of 1A through 1G) 


Indicator 1A. The course or courses, enrichment activities, and/or experiential learning 

opportunities cover key content relevant to working with students with visual 
impairments. 


Indicator IB. The faculty members or other individuals delivering course content were 
qualified for their assigned role. 


Indicator 1C. The product or service adequately addresses all of the research questions, 
hypotheses, or objectives presented. 


Indicator ID. The product or service’s findings are clearly supported by the data or, for a 

concept or position paper or presentation, arguments or theories are backed up 
with valid research and sound reasoning. 


Indicator IE. The analyses conducted are appropriate for the data used in the product or 
service. 


Indicator IF. The product or service’s data sources and instruments are appropriate for 
addressing the research questions, hypotheses, or objectives. 


Indicator 1G. The product or service is easy to read and/or understand, given the intended 
audience. 


DIMENSION 2: PRODUCTS DEEMED OF HIGH RELEVANCE/USEFULNESS TO 
POLICY OR PRACTICE: Synthesis rating (mean of 2A through 2C) 


Indicator 2A. The product or service focuses on topical areas relevant to students with visual 
impairments. 



C-81 


Indicator 2B. The product or service contributes new information to the intended audience on 
the topic being addressed. 

Indicator 2C. The product or service addresses a high-priority need. 

GENERAL COMMENTS 


C-82 




CRITERIA FOR SCORING 


Dimension 1: Products and Services Deemed of High Quality ( Check the number that best describes 
the quality of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1A. The course 

The course or 


The course or 


The course or 


or courses 

courses, 


courses, 


courses, 


cover key 

enrichment 


enrichment 


enrichment 


content. 

activities, 


activities, 


activities, 


enrichment 

and/or 


and/or 


and/or 


activities, 

experiential 


experiential 


experiential 


and/or 

learning 


learning 


learning 


experiential 

opportunities 


opportunities 


opportunities 


learning 

cover key 


cover some 


cover little if 


opportunities 

content 


content 


any content 


relevant to 

relevant to 


relevant to 


relevant to 


working with 

working with 


working with 


working with 


students with 

students with 


students with 


students with 


visual 

visual 


visual 


visual 


impairments. 

impairments. 


impairments 

but omit 

important 

content or 

include 

unnecessary 

content. 


impairments. 



Comments on rating, required for NA: 


C-83 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IB. The faculty 

The faculty 


The faculty 


The faculty 


members or 

members or 


members or 


members or 


other 

other 


other 


other 


individuals 

individuals 


individuals 


individuals 


delivering 

delivering 


delivering 


delivering 


course 

course content 


course content 


course content 


content were 

were highly 


were 


were poorly 


qualified for 

qualified for 


moderately 


qualified for 


their assigned 

their assigned 


qualified for 


their assigned 


role. 

role. 


their assigned 


role. 





role. 




Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

lC.The product 
or service 
adequately 
addresses all 
of the 
research 
questions, 
hypotheses, 
or objectives 
presented. 

All of the 

research 

questions, 

hypotheses, or 

objectives are 

adequately 

addressed. 


Some but not all 
of the research 
questions, 
hypotheses, or 
objectives are 
adequately 
addressed. 


Few if any of 
the research 
questions, 
hypotheses, or 
objectives are 
adequately 
addressed. 


Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 1 

□ 

ID. The product 

All of the 


Some but not all 


Few if any of 


or serviced 

findings are 


of the findings 


the findings are 


findings are 

clearly 


are supported by 


supported by the 


clearly 

supported by the 


the data, or, for 


data, or, for a 


supported by 

data, or, for a 


a concept or 


concept or 


the data, or. 

concept or 


position paper 


position paper 


for a concept 

position paper 


or presentation. 


or presentation, 


or position 

or presentation. 


some but not all 


few if any 


paper or 

arguments or 


arguments or 


arguments or 


presentation. 

theories are 


theories are 


theories are 


arguments or 

backed up with 


backed up with 


backed up with 


theories are 

valid research 


valid research 


valid research 


backed up 

and sound 


and sound 


and sound 


with valid 

reasoning. 


reasoning. 


reasoning. 


research and 







sound 







reasoning. 







Comments on rating, required for NA: 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

IE. The analyses 
conducted are 
appropriate for 
the data used 
in the product 
or service. 

The product 
or service 
uses the best 
and 

appropriate 
analysis 
methods for 
the research 
questions, 
hypotheses, 
or objectives. 


The analysis 
methods are 
reasonable, 
although better 
ones could be 
applied within 
the context. 


The product or 
service uses few 
if any 
appropriate 
methods to 
analyze the 
data; or the 
analytic 

methods are not 
described. 



Comments on rating, required for NA: 


C-85 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IF. The product or 

All of the 


Some but not 


The instruments 


serviced data 

instruments 


all of the 


have little if any 


sources and 

are valid and 


instruments 


validity or 


instruments 

reliable for 


are valid and 


reliability for 


are appropriate 

addressing 


reliable; the 


addressing the 


for addressing 

the research 


validity and/or 


research 


the research 

questions, 


reliability of 


questions, 


questions, 

hypotheses, 


some 


hypotheses, or 


hypotheses, or 

or objectives, 


instruments is 


objectives, and 


objectives. 

and the data 


in question; or 


there are serious 



are collected 


there are some 


problems with 



using sound 


concerns with 


data collection. 



practices. 


the soundness 


or 





of the data 


instrumentation 





collection. 


was applicable 
to the product 
or service but 
was not 

addressed at all. 



Comments on rating, required for NA: 


C-86 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

1G. The product 
or service is 
easy to read 
and 

understand, 
given the 
intended 
audience. 

The product 
or service is 
well written 
throughout 
and can be 
easily 
understood 
by the 
intended 
audience. 


Some but not 
all sections of 
the product or 
service are 
well written, 
and/or the 
information is 
too technical 
for the 
intended 
audience. 


The product 
or service is 
poorly 
written 
throughout. 



Comments on rating, required for NA: 


C-87 


Dimension 2: Products Deemed of High Relevance/Usefulness to Policy or Practice ( Check the 
number that best describes the relevance/usefulness of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2A. The product 

The product 


The product 


The product 


or service 

or service 


or service 


or service 


focuses on 

covers high- 


covers mid- 


doesn’t 


topical areas 

priority 


level- priority 


cover 


in special 

topical areas 


topical areas 


priority 


education 

in special 


in special 


topical areas 


policy or 

education 


education 


in special 


practice 

law, special 


law, special 


education 


relevant to 

education, 


education, or 


law, special 


students with 

or education 


education 


education, or 


visual 

policy 


policy 


education 


impairments. 

relevant to 


relevant to 


policy 



students 


students with 


relevant to 



with visual 


visual 


students 



impairments 


impairments. 


with visual 
impairments. 


Comments on rating, required for NA: 


C-88 


Indicator 

Very High 
Relevance/ 
Usefulness 

High 

Relevance/ 

Usefulness 

Moderate 

Relevance/ 

Usefulness 

Low 

Relevan 

ce/ 

Usefuln 

ess 

Very Low 
Relevance/ 
Usefulness 

Not Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

2B.The product 

The 


The product 


The product or 


or service 

information 


or service is 


service does 


contributes 

presented in 


somewhat 


not provide 


new 

the product 


limited in 


much if any 


information 

or service is 


providing 


new 


to the 

new or 


new 


information or 


intended 

makes a 


information 


does not make 


audience on 

contribution 


or in 


a contribution 


the topic 

to the 


making a 


to the intended 


being 

intended 


contribution 


audience. 


addressed. 

audience. 


to the 







intended 







audience. 





Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 



Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Not Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 1 

□ 

□ 

2C.The product 

The product 


The product 


The product 


or service 

or service 


or service 


or service 


addresses a 

addresses a 


addresses a 


does not 


high-priority 

critical 


less critical 


address a 


need. 

need with 


need with 


need with 



regard to 


regard to 


regard to 



services for 


services for 


services for 



students 


students 


students 



with visual 


with visual 


with visual 



impairment 


impairment 


impairment 



s. 


s. 


s. 


Comments on rating, required for NA: 


C-89 


NIUSI-PLAN LEADSCAPE 


INDICATORS AND SCORING RUBRIC FOR THE 

NATIONAL INSTITUTE FOR URBAN SCHOOL IMPROVEMENT PRINCIPAL LEADERSHIP 

ACADEMIES INITIATIVE 

(CFDA 84.325P) 


Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: PRODUCTS AND SERVICES DEEMED OF HIGH QUALITY: 
Synthesis rating (mean of 1A through 11) 


Indicator 1A. The content of the product or service is based on rigorous and relevant 

evidence-based practices and on scientifically based research, when available. 


Indicator IB. The product or service uses an approach to technical assistance or professional 
development that is appropriate for promoting the desired change in policy or 
practice. 


Indicator 1C. The product or service assists school principals in the development and 

implementation of school improvement strategies with regard to 1) access to and 
2) participation and progress in the general education curriculum in the least 
restrictive environment (LRE) for students with disabilities. 


Indicator ID. The product or service assists school principals in continuous improvement and 
sustainability of school improvement strategies with regard to 1) access to and 
2) participation and progress in the general education curriculum in the least 
restrictive environment (LRE) for students with disabilities. 


Indicator IE. The product or service adequately addresses all of the research questions, 
hypotheses, or objectives presented. 


Indicator IF. The product or service’s findings are clearly supported by the data, or, for a 

concept or position paper or presentation, arguments or theories are backed up 
with valid research and sound reasoning. 


Indicator 1G. The analyses conducted are appropriate for the data used in the product or 
service. 
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Indicator 1H. The product or service’s data sources and instruments are appropriate for 
addressing the research questions, hypotheses, or objectives. 


Indicator 11. The product or service is easy to read and/or understand, given the intended 
audience. 




DIMENSION 2: PRODUCTS DEEMED OF HIGH RELEVANCE/USEFULNESS TO 
POLICY OR PRACTICE: Synthesis rating (mean of 2A through 2D) 


Indicator 2A. The product or service builds the capacity of principals to use evidence-based 
school improvement practices. 


Indicator 2B. The product or service supports the scaling -up of school improvement strategies 
that foster 1) access to and 2) participation and progress in the general education 
curriculum in the LRE for students with disabilities. 


Indicator 2C. The product or service takes into account the state and local conditions in which 
school improvement must occur. 


Indicator 2D. The product or service contributes new information to the intended audience on 
the topic being addressed. 


GENERAL COMMENTS 


C-91 




CRITERIA FOR SCORING 


Dimension 1: Products and Services Deemed of High Quality ( Check the number that best describes 
the quality of the product or sendee) 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

1A. The content 
of the product 
or service is 
based on 
rigorous and 
relevant 
evidence- 
based 

practices and 
on 

scientifically 

based 

research, 

when 

available. 

In developing 
the product or 
service, the 
grantee 

consistently used 
relevant and 
rigorous 
research with 
appropriate 
designs, 
methods, 
measures, and 
analyses over 
research with 
less relevance or 
weaker designs. 


In developing the 
product or service, 
the grantee 
sometimes used 
relevant and 
rigorous research 
with appropriate 
designs, methods, 
measures, and 
analyses over 
those with weaker 
designs. 


In developing 
the product or 
service, the 
grantee rarely if 
ever used 
relevant and 
rigorous 
research with 
appropriate 
designs, 
methods, 
measures, or 
analyses. 



Comments on rating, required for NA: 


C-92 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IB. The product 

The product or 


The product or 


The product or 


or service uses 

service uses an 


service uses an 


service uses an 


an approach to 

approach to 


approach to 


approach to 


technical 

technical 


technical 


technical 


assistance or 

assistance or 


assistance or 


assistance or 


professional 

professional 


professional 


professional 


development 

development 


development that 


development 


that is 

that is highly 


is moderately 


that is not very 


appropriate 

appropriate for 


appropriate for 


appropriate for 


for promoting 

promoting the 


promoting the 


promoting the 


the desired 

desired change 


desired change in 


desired change 


change in 

in policy or 


policy or practice, 


in policy or 


policy or 

practice. 


but better 


practice. 


practice. 



approaches could 







have been used. 





Comments on rating, required for NA: 


C-93 



Very High 

High 


Low 


Not 


Quality 

Quality 

Moderate Quality 

Quality 

Very Low Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1C. The product or 

The product or 


The product or 


The product or 


service assists 

service provides 


service provides 


service provides 


school principals 

much assistance to 


some assistance to 


little assistance to 


in the 

school principals in 


school principals 


school principals in 


development and 

the development 


in the development 


EITHER the 


implementation 

AND 


AND 


development OR 


of school 

implementation of 


implementation of 


implementation of 


improvement 

school 


school 


school improvement 


strategies with 

improvement 


improvement 


strategies with 


regard to 1) 

strategies with 


strategies with 


regard to 1) access 


access to and 2) 

regard to 1) access 


regard to 1) access 


to and 2) 


participation and 

to and 2) 


to and 2) 


participation and 


progress in the 

participation and 


participation and 


progress in the 


general 

progress in the 


progress in the 


general education 


education 

general education 


general education 


curriculum in the 


curriculum in the 

curriculum in the 


curriculum in the 


LRE for students 


LRE for students 

LRE for students 


LRE for students 


with disabilities. 


with disabilities. 

with disabilities. 


with disabilities. 




Comments on rating, required for NA: 


C-94 



Very High 

High 


Low 


Not 


Quality 

Quality 

Moderate Quality 

Quality 

Very Low Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

ID. The product or 

The product or 


The product or 


The product or 


service assists 

service provides 


service provides 


service provides 


school principals 

much assistance to 


some assistance to 


little assistance to 


in continuous 

school principals 


school principals 


school principals in 


improvement and 

in the continuous 


in the continuous 


the continuous 


sustainability of 

improvement and 


improvement and 


improvement and 


school 

sustainability of 


sustainability of 


sustainability of 


improvement 

school 


school 


school 


strategies with 

improvement 


improvement 


improvement 


regard to 1) access 

strategies with 


strategies with 


strategies with 


to and 2) 

regard to 1) access 


regard to 1) access 


regard to 1) access 


participation and 

to and 2) 


to and 2) 


to and 2) 


progress in the 

participation and 


participation and 


participation and 


general education 

progress in the 


progress in the 


progress in the 


curriculum in the 

general education 


general education 


general education 


LRE for students 

curriculum in the 


curriculum in the 


curriculum in the 


with disabilities. 

LRE for students 


LRE for students 


LRE for students 



with disabilities. 


with disabilities. 


with disabilities. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IE. The product or 
service 
adequately 
addresses all of 
the research 
questions, 
hypotheses, or 
objectives 
presented. 

All of the 

research 

questions, 

hypotheses, 

or 

objectives 

are 

adequately 

addressed. 


Some but 
not all of 
the research 
questions, 
hypotheses, 
or 

objectives 

are 

adequately 

addressed. 


Few if any of 
the research 
questions, 
hypotheses, or 
objectives are 
adequately 
addressed. 


Comments on rating, required for NA: 






C-95 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ I 

□ 

□ 

IF. The product or 

All of the 


Some but 


Few if any of the 


service’s 

findings are 


not all of 


findings are 


findings are 

clearly 


the findings 


supported by the 


clearly 

supported 


are clearly 


data, or, for a 


supported by 

by the data 


supported 


concept or 


the data or, for 

or, for a 


by the data, 


position paper, 


a concept or 

concept or 


or, for a 


few if any of the 


position paper 

position 


concept or 


arguments or 


or presentation. 

paper, all 


position 


theories are 


arguments or 

arguments 


paper, some 


backed up with 


theories are 

or theories 


but not all 


valid research or 


backed up with 

are backed 


of the 


sound reasoning. 


valid research 

up with 


arguments 




and sound 

valid 


or theories 




reasoning. 

research 


are backed 





and sound 


up with 





reasoning. 


valid 







research and 







sound 







reasoning. 




Comments on rating, required for NA: 


C-96 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1G. The analyses 

The product 


The analysis 


The product or 


conducted are 

or service 


methods are 


service uses few 


appropriate 

uses the best 


reasonable, 


if any 


for the data 

appropriate 


although better 


appropriate 


used in the 

analysis 


ones could be 


methods to 


product or 

methods for 


applied within 


analyze the data 


service. 

the research 


the context. 


or the analytic 



questions, 




methods are not 



hypotheses, 

or 

objectives. 




described. 


Comments on rating, required for NA: 


C-97 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1H. The product 

All of the 


Some but not 


The instruments 


or service’s 

instruments 


all of the 


have little if any 


data sources 

are valid and 


instruments are 


validity or 


and 

reliable for 


valid and 


reliability for 


instruments 

addressing 


reliable; the 


addressing the 


are 

the research 


validity and/or 


research 


appropriate 

questions, 


reliability of 


questions, 


for 

hypotheses, 


some 


hypotheses, or 


addressing 

or 


instruments is 


objectives, and 


the research 

objectives, 


in question; or 


there are serious 


questions. 

and the data 


there are some 


problems with 


hypotheses, 

are collected 


concerns with 


data collection. 


or objectives. 

using sound 


the soundness 


or 



practices. 


of the data 


instrumentation 





collection. 


was applicable 
to the paper or 
presentation but 
was not 

addressed at all. 



Comments on rating, required for NA: 


C-98 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

11. The product 

The product 


Some but not 


The product 


or service is 

or service is 


all sections of 


or service is 


easy to read 

well written 


the product or 


poorly 


and 

throughout 


service are 


written 


understand, 

and can be 


well written. 


throughout. 


given the 

easily 


and/or the 




intended 

understood 


information is 




audience. 

by the 


too technical 





intended 


for the 





audience. 


intended 







audience. 




Comments on rating, required for NA: 


C-99 


Dimension 2: Products Deemed of High Relevance/Usefulness to Policy or Practice ( Check the 
number that best describes the relevance/usefulness of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2A. The product or 

The product or 


The product or 


It is not clear 


service builds 

service 


service provides 


how the product 


the capacity of 

provides 


some assistance 


or service will 


principals to 

extensive 


to principals in 


help principals 


use evidence- 

assistance to 


using evidence- 


to use evidence- 


based school 

principals in 


based school- 


based school- 


improvement 

using 


improvement 


improvement 


practices. 

evidence- 


practices but is 


practices. 



based school- 


not designed 





improvement 


with that specific 





practices. 


purpose in mind. 




Comments on rating, required for NA: 


C-100 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B.The product or 

The product or 


The product or 


The product or 


service supports 

service 


service supports 


service supports 


the scaling-up 

supports the 


school 


school 


of school 

scaling-up of 


improvement 


improvement 


improvement 

school 


strategies that 


strategies that 


strategies that 

improvement 


foster 1) access 


foster 1) access 


foster 1) access 

strategies that 


to and 2) 


to and 2) 


to and 2) 

foster 1) 


participation and 


participation 


participation 

access to and 


progress in the 


and progress in 


and progress in 

2) 


general 


the general 


the general 

participation 


education 


education 


education 

and progress 


curriculum in the 


curriculum in 


curriculum in 

in the general 


LRE for students 


the LRE for 


the LRE for 

education 


with disabilities 


students with 


students with 

curriculum in 


BUT it would be 


disabilities BUT 


disabilities. 

the LRE for 


moderately 


would be 



students with 


difficult to scale 


VERY difficult 



disabilities. 


up such a service 


to scale up such 





or support. 


a service or 







support. 


Comments on rating, required for NA: 


C-101 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2C.The product 

The product or 


The product 


The product 


or service 

service takes 


or service 


or service 


takes into 

into account 


recognizes 


does not 


account the 

the variation 


the variation 


recognize or 


state and local 

in local 


in local 


take into 


conditions in 

conditions 


conditions 


account the 


which school 

under which 


under which 


variation in 


improvement 

school 


school 


local 


must occur. 

improvement 


improvement 


conditions 



must occur. 


must occur 


under which 





but does not 


school 





take them into 


improvement 





account fully. 


must occur. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2D. The product 

The 


The product 


The product 


or service 

information 


or service is 


or service 


contributes 

presented in 


somewhat 


does not 


new 

the product or 


limited in 


provide much 


information to 

service is new 


providing 


if any new 


the intended 

or makes a 


new 


information 


audience on 

contribution to 


information 


or does not 


the topic 

the intended 


or in making 


make a 


being 

audience. 


a contribution 


contribution 


addressed. 



to the 


to the 





intended 


intended 





audience. 


audience. 


Comments on rating, required for NA: 


C-102 


NPDCI 

INDICATORS AND SCORING RUBRIC FOR THE 
NATIONAL PROFESSIONAL DEVELOPMENT CENTER ON INCLUSION 

(CFDA 84.325S) 


Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: PRODUCTS AND SERVICES DEEMED OF HIGH QUALITY: 
Synthesis rating (mean of 1A through 1H) 




Indicator 1A. The content of the product or service is based on rigorous and relevant 

evidence -based practices and on scientifically based research, when available. 


Indicator IB. The product or service uses an approach to technical assistance or professional 
development that is appropriate for promoting the desired change in policy or 
practice. 


Indicator 1C. The product or service reflects current theory and research on adult learning, 
including motivation, reinforcement, retention, transference, and use of 
reflective practice. 


Indicator ID. The product or service adequately addresses all of the research questions, 
hypotheses, or objectives presented. 


Indicator IE. The product or service’s findings are clearly supported by the data, or, for a 

concept or position paper or presentation, arguments or theories are backed up 
with valid research and sound reasoning. 


Indicator IF. The analyses conducted are appropriate for the data used in the paper or 
presentation. 


Indicator 1G. The product or service’s data sources, instruments, or information gathering 

processes are appropriate for addressing the research questions, hypotheses, or 
objectives. 


Indicator 1H. The product or service is easy to read and/or understand, given the intended 
audience. 



C-103 


DIMENSION 2: PRODUCTS DEEMED OF HIGH RELEVANCE/USEFULNESS TO 
POLICY OR PRACTICE: Synthesis rating (mean of 2A through 2F) 


Indicator 2A. The product or service addresses an identified need with regard to inclusion in 
the preschool setting. 


Indicator 2B. The product or service helps recipients to meet state personnel standards or 
other requirements in state law or regulation for serving children with 
disabilities or infants and toddlers with disabilities. 


Indicator 2C. The product or service takes into account the local conditions in which 
preschool inclusion occurs. 


Indicator 2D. The product or service is suitable for dissemination to a wide variety of 
audiences. 


Indicator 2E. The product or service will assist early childhood technical assistance providers 
in developing training materials. 


Indicator 2F. The paper or presentation contributes new information on the topic being 
addressed. 


GENERAL COMMENTS 


C-104 




CRITERIA FOR SCORING 


Dimension 1: Products and Services Deemed of High Quality ( Check the number that best describes 
the quality of the product or sendee) 



Very High 

High 


Low 

Very Low 

Not 


Quality 

Quality 

Moderate Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1A. The content of the 

In developing 


In developing the 


In developing the 


product or service 

the product or 


product or service, 


product or 


is based on 

service, the 


the grantee 


service, the 


rigorous and 

grantee 


sometimes used 


grantee rarely if 


relevant evidence- 

consistently 


relevant and 


ever used 


based practices and 

used relevant 


rigorous research 


relevant and 


on scientifically 

and rigorous 


with appropriate 


rigorous research 


based research, 

research with 


designs, methods, 


with appropriate 


when available. 

appropriate 


measures, and 


designs. 



designs, 


analyses over those 


methods, 



methods, 


with weaker 


measures, or 



measures, and 
analyses over 
research with 
less relevance 
or weaker 
designs. 


designs. 


analyses. 


Comments on rating, required for NA: 


C-105 



Very High 

High 


Low 

Very Low 

Not 


Quality 

Quality 

Moderate Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IB. The product or 

The product or 


The product or 


The product or 


service uses an 

service uses an 


service uses an 


service uses an 


approach to 

approach to 


approach to 


approach to 


technical assistance 

technical 


technical assistance 


technical 


or professional 

assistance or 


or professional 


assistance or 


development that is 

professional 


development that is 


professional 


appropriate for 

development 


moderately 


development that 


promoting the 

that is highly 


appropriate for 


is not very 


desired change in 

appropriate for 


promoting the 


appropriate for 


policy or practice. 

promoting the 


desired change in 


promoting the 



desired change 


policy or practice, 


desired change in 



in policy or 


but better 


policy or 



practice. 


approaches could 


practice. 





have been used. 





Comments on rating, required for NA: 


C-106 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1C. The product or 

The product or 


The product or 


The product 


service reflects 

service 


service 


or service 


current theory 

accurately 


accurately 


reflects few 


and research 

reflects all 


reflects some 


if any 


on adult 

relevant 


but not all 


components 


learning. 

components of 


components of 


of adult 


including 

adult learning 


adult learning 


learning 


motivation, 

theory. 


theory, 


theory. 


reinforcement. 

including 


including 




retention. 

motivation. 


motivation, 




transference, 

reinforcement, 


reinforcement, 




and use of 

retention. 


retention, 




reflective 

transference of 


transference of 




practice. 

content, and 


content, and 





facilitation of 


facilitation of 





reflective 


reflective 





practice. 


practice. 




Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

ID. The product 

All of the 


Some but not 


Few if any of 


or service 

research 


all of the 


the research 


adequately 

questions, 


research 


questions, 


addresses all 

hypotheses, or 


questions, 


hypotheses, 


of the research 

objectives are 


hypotheses, or 


or objectives 


questions. 

adequately 


objectives are 


are 


hypotheses, or 

addressed. 


adequately 


adequately 


objectives 

presented. 



addressed. 


addressed. 


Comments on rating, required for NA: 


C-107 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IE. The product or 

All of the 


Some but not 


Few if any of 


service’s 

findings are 


all of the 


the findings 


findings are 

clearly 


findings are 


are supported 


clearly supported 

supported by 


clearly 


by the data, 


by the data or, 

the data, or, 


supported by 


or, for a 


for a concept or 

for a 


the data, or, 


concept or 


position paper or 

concept or 


for a concept 


position 


presentation. 

position 


or position 


paper, few if 


arguments or 

paper, all 


paper, some 


any of the 


theories are 

arguments 


but not all of 


arguments or 


backed up with 

or theories 


the arguments 


theories are 


valid research 

are backed 


or theories are 


backed up 


and sound 

up with 


backed up 


with valid 


reasoning. 

valid 


with valid 


research or 



research and 


research and 


sound 



sound 


sound 


reasoning. 



reasoning. 


reasoning. 




Comments on rating, required for NA: 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

IF. The analyses 
conducted are 
appropriate for 
the data used in 
the product or 
service. 

The product 
or service 
uses the best 
appropriate 
analysis 
methods for 
the research 
questions, 
hypotheses, 
or 

objectives. 


The analysis 
methods are 
reasonable, 
although 
better ones 
could be 
applied 
within the 
context. 


The product 
or service 
uses few if 
any 

appropriate 
methods to 
analyze the 
data, or the 
analytic 
methods are 
not described. 



Comments on rating, required for NA: 


C-108 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1G. The product 

All of the 


Some but not 


The instruments 


or service’s 

instruments 


all of the 


have little if any 


data sources 

are valid and 


instruments 


validity or 


and 

reliable for 


are valid and 


reliability for 


instruments 

addressing 


reliable; the 


addressing the 


are appropriate 

the research 


validity and/or 


research 


for addressing 

questions, 


reliability of 


questions, 


the research 

hypotheses, 


some 


hypotheses, or 


questions, 

or 


instruments is 


objectives, and 


hypotheses, or 

objectives, 


in question; or 


there are serious 


objectives. 

and the data 


there are some 


problems with 



are collected 


concerns with 


data collection, or 



using sound 


the soundness 


instrumentation 



practices. 


of the data 


was applicable to 





collection. 


the paper or 
presentation but 
was not 

addressed at all. 



Comments on rating, required for NA: 


C-109 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

1H. The paper or 
presentation is 
easy to read 
and 

understand, 
given the 
intended 
audience. 

The product 
or service is 
well written 
throughout 
and can be 
easily 
understood 
by the 
intended 
audience. 


Some but not 
all sections of 
the product or 
service are 
well written, 
and/or the 
information is 
too technical 
for the 
intended 
audience. 


The product or 
service is poorly 
written 
throughout. 



Comments on rating, required for NA: 


C-110 


Dimension 2: Products Deemed of High Relevance to Policy or Practice ( Check the number that best 
describes the relevance/usefulness of the product or service ) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2A.The product or 

The product or 


The product or 


The product or 


service 

service addresses 


service 


service does not 


addresses an 

critical personnel 


addresses 


address critical 


identified need 

development 


secondary 


or secondary 


with regard to 

needs or an 


personnel 


personnel 


inclusion in the 

important gap or 


development 


development 


preschool 

weakness in 


needs or a less 


needs or gaps or 


setting. 

services, 


important gap or 


weakness in 



infrastructure, or 


weakness in 


services, 



opportunities. 


services, 


infrastructure, or 





infrastructure, or 
opportunities. 


opportunities. 


Comments on rating, required for NA: 


C-lll 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B.The product or 

The product or 


The product or 


The product or 


service helps 

service directly 


service 


service does not 


recipients to 

helps recipients 


indirectly helps 


help recipients 


meet state 

to meet state 


recipients to 


to meet any 


personnel 

personnel 


meet state 


state personnel 


standards or 

standards or other 


personnel 


standards or 


other 

requirements in 


standards or 


other 


requirements in 

state law or 


other 


requirements in 


state law or 

regulation for 


requirements in 


state law or 


regulation for 

serving children 


state law or 


regulation for 


serving children 

with disabilities, 


regulation for 


serving children 


with disabilities 

and the 


serving children 


with 


or infants and 

connections 


with disabilities, 


disabilities. 


toddlers with 

between the 


but the 




disabilities. 

preparation/ 


connections 





training and state 


between the 





standards are 


preparation/ 





explicit. 


training and 
state standards 
are not explicit. 




Comments on rating, required for NA: 


C- 112 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2C.The product or 

The product or 


The 


The product 


service takes 

service takes 


product or 


or service 


into account 

into account the 


service 


does not 


the local 

local conditions 


recognizes 


recognize or 


conditions in 

under which 


the local 


take into 


which 

preschool 


conditions 


account the 


preschool 

inclusion 


under 


local 


inclusion 

occurs. 


which 


conditions 


occurs. 



preschool 


under which 





inclusion 


preschool 





occurs, but 


inclusion 





does not 
take them 
into 

account 

fully. 


occurs. 


Comments on rating, required for NA: 


C-113 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2D. The product 

The product or 


The 


The product 


or service is 

service is 


product or 


or service is 


suitable for 

suitable for 


service is 


not suitable 


dissemination 

dissemination 


suitable for 


for 


to a wide 

to a wide 


dissemina- 


dissemina- 


variety of 

variety of 


tion to 


tion at all or 


audiences. 

audiences. 


some but 


is only 





not a wide 


suitable for 





variety of 


dissemina- 





audiences. 


tion to one 
narrowly 
defined 
audience. 


Comments on rating, required for NA: 


C- 114 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2E.The product or 

The product or 


The 


The product 


service will 

service will 


product or 


or service 


assist early 

provide direct. 


service will 


will not 


childhood 

extensive 


provide 


assist early 


technical 

assistance to 


indirect 


childhood 


assistance 

early childhood 


assistance 


technical 


providers in 

technical 


or limited 


assistance 


developing 

assistance 


direct 


providers in 


training 

providers in 


assistance 


developing 


materials. 

developing 


to early 


training 



training 


childhood 


materials. 



materials. 


technical 

assistance 

providers 

in 

developing 

training 

materials. 




Comments on rating, required for NA: 


C- 115 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2F.The product or 

The 


The product 


The product 


service 

information 


or service is 


or service 


contributes new 

presented in 


somewhat 


does not 


information on 

the product 


limited in 


provide 


the topic being 

or service is 


providing 


much if any 


addressed. 

new or 


new 


new 



makes a 


information 


information 



contribution 


or in 


or does not 



to the 


making a 


make a 



intended 


contribution 


contribution 



audience. 


to the 


to the 





intended 


intended 





audience. 


audience. 


Comments on rating, required for NA: 
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PDA 


INDICATORS AND SCORING RUBRIC FOR THE 
PROFESSIONAL DEVELOPMENT IN AUTISM CENTER 
(CFDA 84.325G) 


Product: 

Reviewer: Date: 


SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: PRODUCTS AND SERVICES DEEMED OF HIGH QUALITY: 
Synthesis rating (mean of 1A through 1G) 


Indicator 1A. The content of the product or service is based on rigorous and relevant 

evidence-based practices and on scientifically based research, when available. 


Indicator IB. The product or service reflects current theory and research on adult learning, 
including motivation, reinforcement, retention, transference, and use of 
reflective practice. 


Indicator 1C. The product or service adequately addresses all of the research questions, 
hypotheses, or objectives presented. 


Indicator ID. The product or service’s data sources and instruments are appropriate for 
addressing the research questions, hypotheses, or objectives. 


Indicator IE. The analyses conducted are appropriate for the data used in the product or 
service. 


Indicator IF. The product or service’s findings are clearly supported by the data, or, for a 

concept or position paper or presentation, arguments or theories are backed up 
with valid research and sound reasoning. 


Indicator 1G. The product or service is easy to read and/or understand, given the intended 
audience. 


DIMENSION 2: PRODUCTS DEEMED OF HIGH RELEVANCE/USEFULNESS TO 
POLICY OR PRACTICE: Synthesis rating (mean of 2A through 2F) 


Indicator 2A. The product or service addresses a high-priority need. 
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Indicator 2B. The product or service provides opportunities for trainees to see and engage in 
the identified methods and practices in authentic settings. 


Indicator 2C. The product or service is accessible to all targeted groups. 


Indicator 2D. The product or service includes a provision for technical assistance, for 
example, phone support or on-line help. 


Indicator 2E. The product or service takes into account local conditions in which services for 
children with autism and their families are provided. 


Indicator 2F. The product or service contributes new information to the intended audience on 
the topic being addressed. 


GENERAL COMMENTS 
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CRITERIA FOR SCORING 


Dimension 1: Products and Services Deemed of High Quality ( Check the number that best describes 
the quality of the product or sendee) 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1A. The content of 

In developing the 


In developing the 


In 


the product or 

product or 


product or 


developing 


service is based 

service, the 


service, the 


the product 


on rigorous and 

grantee 


grantee 


or service, 


relevant 

consistently used 


sometimes used 


the grantee 


evidence-based 

relevant and 


relevant and 


consistently 


practices and on 

rigorous research 


rigorous research 


did not use 


scientifically 

with appropriate 


with appropriate 


relevant and 


based research. 

designs. 


designs, 


rigorous 


when available. 

methods, 


methods, 


research 



measures, and 


measures, and 


with 



analyses over 


analyses over 


appropriate 



research with 


those with 


designs, 



less relevance or 


weaker designs. 


methods, 



weaker designs. 




measures, or 







analyses. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

IB. The product or 

The product or 


The product or 


The product 


service reflects 

service 


service 


or service 


current theory 

accurately 


accurately 


does not 


and research on 

reflects all 


reflects some but 


accurately 


adult learning. 

relevant 


not all 


reflect 


including 

components of 


components of 


components 


motivation, 

adult learning 


adult learning 


of adult 


reinforcement. 

theory, including 


theory, including 


learning 


retention, 

motivation. 


motivation, 


theory. 


transference, 

reinforcement. 


reinforcement. 




and use of 

retention. 


retention. 




reflective 

transference of 


transference of 




practice. 

content, and 


content, and 





facilitation of 


facilitation of 





reflective 


reflective 





practice. 


practice. 





Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1C. The product or 
service 
adequately 
addresses all 
of the research 
questions, 
hypotheses, or 
objectives 
presented. 

All of the 

research 

questions, 

hypotheses, 

or objectives 

are 

adequately 

addressed. 


Some but not 
all of the 
research 
questions, 
hypotheses, or 
objectives are 
adequately 
addressed. 


Few if any of the 

research 

questions, 

hypotheses, or 

objectives are 

adequately 

addressed. 


Comments on rating, required for NA: 







Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

ID. The product 

All of the 


Some but not 


The instruments 


or service’s 

instruments 


all of the 


have little if any 


data sources 

are valid and 


instruments 


validity or 


and 

reliable for 


are valid and 


reliability for 


instruments 

addressing 


reliable; the 


addressing the 


are appropriate 

the research 


validity 


research 


for addressing 

questions, 


and/or 


questions, 


the research 

hypotheses, 


reliability of 


hypotheses, or 


questions, 

or 


some 


objectives, and 


hypotheses, or 

objectives, 


instruments is 


there are serious 


objectives. 

and the data 


in question; or 


problems with 



are collected 


there are some 


data collection. 



using sound 


concerns with 


or 



practices. 


the soundness 


instrumentation 





of the data 


was applicable to 





collection. 


the product or 
service but was 
not addressed at 

all. 



Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 1 

□ 

□ 

□ 

□ 

IE. The analyses 
conducted are 
appropriate for 
the data used 
in the product 
or service. 

The product 
or service 
uses the best 
appropriate 
analysis 
methods for 
the research 
questions, 
hypotheses, 
or objectives. 


The analysis 
methods are 
reasonable, 
although 
better ones 
could be 
applied 
within the 
context. 


The study 
uses few if 
any 

appropriate 
methods to 
analyze the 
data, or the 
analytic 
methods are 
not 

described. 


Comments on rating, required for NA: 







Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ X 

□ 

□ 

□ 

□ 

IF. The product or 

All of the 


Some but not 


Few if any of 


service’s 

findings are 


all of the 


the findings 


findings are 

clearly 


findings are 


are 


clearly 

supported by 


clearly 


supported by 


supported by 

the data or, 


supported by 


the data, or, 


the data or, for 

for a concept 


the data, or, 


for a concept 


a concept or 

or position 


for a concept 


or position 


position paper 

paper, all 


or position 


paper, few if 


or 

arguments or 


paper, some 


any of the 


presentation, 

theories are 


but not all of 


arguments or 


arguments or 

backed up 


the 


theories are 


theories are 

with valid 


arguments or 


backed up 


backed up 

research and 


theories are 


with valid 


with valid 

sound 


backed up 


research or 


research and 

reasoning. 


with valid 


sound 


sound 



research and 


reasoning. 


reasoning. 



sound 







reasoning. 




Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 

Not 


Quality 

Quality 

Quality 

Quality 

Quality 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

1G. The product 
or service is 
easy to read 
and 

understand, 
given the 
intended 
audience. 

The product 
or service is 
well written 
throughout 
and can be 
easily 
understood 
by the 
intended 
audience. 


Some but not 
all sections of 
the product or 
service are 
well written, 
and/or the 
information is 
too technical 
for the 
intended 
audience. 


The product 
or service is 
poorly 
written 
throughout. 


Comments on rating, required for NA: 
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Dimension 2: Products Deemed of High Relevance/Usefulness to Policy or Practice ( Check the 
number that best describes the relevance/usefulness of the product or sendee ) 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2A. The product 

The product or 


The product or 


The product or 


or service 

service 


service 


service does 


addresses a 

addresses 


addresses 


not address 


high-priority 

critical 


secondary 


critical or 


need. 

personnel 


personnel 


secondary 



development 


development 


personnel 



needs or an 


needs or a less 


development 



important gap 


important gap 


needs or gaps 



or weakness in 


or weakness in 


or weakness in 



services, 


services, 


services, 



infrastructure, 


infrastructure, 


infrastructure, 



or 


or 


or 



opportunities 


opportunities. 


opportunities. 


Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2B.The product or 

The product or 


The product or 


The product or 


service 

service 


service 


service does 


provides 

provides 


provides 


not provide 


opportunities 

ample 


limited 


opportunities 


for trainees to 

opportunities 


opportunities 


for teams to 


see and 

for teams to 


for teams to 


observe or 


engage in the 

observe and/or 


observe and/or 


apply the 


identified 

apply the 


apply the 


identified 


methods and 

identified 


identified 


methods and 


practices in 

methods and 


methods and 


practices in 


authentic 

practices in 


practices in 


authentic 


settings. 

authentic 


authentic 


settings, such 



settings, such 


settings, such 


as children’s 



as children’s 


as children’s 


homes, day 



homes, day 


homes, day 


care facilities, 



care facilities, 


care facilities, 


and schools. 



and schools. 


and schools. 




Comments on rating, required for NA: 
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Indicator 

Very High 
Relevance/ 
Usefulness 

High 

Relevance/ 

Usefulness 

Moderate 

Relevance/ 

Usefulness 

Low 

Relevance/ 

Usefulness 

Very Low 
Relevance/ 
Usefulness 

Not 

Applicable 

5 

□ 

4 

□ 

3 

□ 

2 

□ 

1 

□ 

NA 

□ 

2C.The product 
or service is 
accessible to 
all targeted 
groups. 

The product 
or service is 
easy for 
targeted 
groups to 
obtain and 
use, when 
needed. 


Although it 
may be 
available, 
the product 
or service is 
not easy for 
targeted 
groups to 
obtain or 
use, when 
needed. 


Few if any 
targeted 
groups can 
obtain or use 
the product 
or service 
when 
needed. 



Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2D. The product 

Technical 


Technical 


Technical 


or service 

assistance is 


assistance is 


assistance is 


includes a 

easily 


moderately 


very difficult 


provision for 

accessible to 


accessible 


to access for 


technical 

potential 


to potential 


potential 


assistance, for 

consumers, 


consumers 


consumers, 


example, 

and is “user- 


or is only 


or the 


phone 

friendly.” 


moderately 


technical 


support or on- 



“user- 


assistance is 


line help. 



friendly.” 


difficult to 







use. 


Comments on rating, required for NA: 
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Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2E.The product 

The product or 


The product 


The product 


or service 

service is 


or service is 


or service is 


takes into 

sufficiently 


robust enough 


not robust 


account local 

robust to 


to achieve its 


enough to 


conditions in 

achieve its 


professional 


achieve its 


which 

professional 


development 


professional 


services for 

development 


aims under 


development 


children with 

aims under 


some local 


aims under 


autism and 

most or all local 


conditions 


varying local 


their families 

conditions and 


and ultimately 


conditions or 


are provided. 

ultimately to 


to achieve its 


to achieve its 



achieve its aims 


aims 


aims for 



regarding the 


regarding the 


service 



delivery of 


delivery of 


delivery to 



services to 


services to 


children with 



children with 


children with 


autism under 



autism under 


autism under 


varying local 



most or all local 


some local 


conditions. 



conditions. 


conditions. 




Comments on rating, required for NA: 



Very High 

High 

Moderate 

Low 

Very Low 



Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Not 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Applicable 


5 

4 

3 

2 

1 

NA 

Indicator 

□ 

□ 

□ 

□ 

□ 

□ 

2F. The product 

The 


The product 


The product 


or service 

information 


or service is 


or service 


contributes 

presented in the 


somewhat 


does not 


new 

product or 


limited in 


provide much 


information 

service is new 


providing new 


if any new 


to the 

or makes a 


information or 


information or 


intended 

contribution to 


in making a 


does not make 


audience on 

the intended 


contribution 


a contribution 


the topic 

audience. 


to the 


to the 


being 



intended 


intended 


addressed. 



audience. 


audience. 


Comments on rating, required for NA: 


C-125 


Appendix D 


Institution of Higher Education Course of Study Survey 


DIRECTIONS: The focus of this survey is the course of study for which you requested Personnel 

Development Program (PDP) funding in FY [pre-filled with application year]. For the purposes of this 
survey, please think of a course of study as a set of courses to prepare candidates to perform a particular 
professional or paraprofessional role; this set of courses almost always results in a degree, a particular 
credential (i.e., license, certificate, or endorsement), or both. The setting of courses is another defining 
aspect of the course of study (e.g., on a particular campus, online). 

The box below provides identifying information about the course of study for which you requested PDP 
funding. Westat staff extracted this information from your grant application. 


Application/Award Number: [pre-filled] 

PDP Priority/Focus Area: [pre-filled] 

Project Title: [pre-filled] 

Roles for which candidates were being prepared: [pre-filled] 
Disability area: [pre-filled] 

Degree(s) for which candidates were being prepared: [pre-filled] 
Credential(s) for which candidates were being prepared: [pre-filled] 
Setting: [pre-filled] 
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Question 1 


Text 

At the time of your grant application, in what way was the [predominant] 1 course of Priority = all 

study for which you requested funding related to an existing course of study? Select 

one. Funding = all 




Response Options 

Go to 


1 Our grant application was intended to support or expand an existing Question 3, if 

course of study, with essentially the same central purpose. That is, the Funded 

[predominant] course of study for which we requested funding was in 
existence at the time of our application. 

Question 5, if 
Non-funded 


2 


Our grant application was intended to fund a course of study that was built 
upon one or more existing courses of study but that differed from those 
existing courses of study in at least one of the following features: the role 
or roles for which candidates are prepared, the degrees or credentials 
candidates seek, or the setting. That is, the [predominant] course of study 
for which we requested funding differed substantially from any existing 
course of study, but it was not completely new. 


Question 5, if 
Funded 


Question 2, if 
Non-funded 


3 The [predominant] course of study for which we requested funding was Question 5, if 
not in existence at the time of our application, nor was it built upon an Funded 

existing predominant course of study. 

Question 2, if 
Non-funded 


1 The word [predominant] appears in the on-screen survey only when more than one course of study are 
included in the survey. 
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Question 2 


Text 


Condition 

The [predominant] course of study that your grant application was intended to develop 
was new or differed substantially from any existing course of study. The application 
was not funded. Which of the following occurred? Select one. 

Priority = all 

Funding = 
Non-Funded 


Response Options 

Go to 

1 

The [predominant] course of study was not developed. 

End, if single 
COS 



Question 27, if 
multiple COSs 

2 

The [predominant] course of study was developed using PDP funding 
from a different grant. 

Question 5 

3 

External sources of funding were used to develop the [predominant] 
course of study. The amount of external funding was equal to or greater 
than the amount requested for the course of study in our PDP grant 
application. 

Question 5 

4 

External sources of funding were used to develop the [predominant] 
course of study. The amount of external funding was smaller than the 
amount requested for that course of study in our PDP grant application. 

Question 4 

5 

The [predominant] course of study was developed with no external 
funding. 

Question 4 
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Question 3 


Text 


Condition 

The [predominant] course of study that your grant application was intended to support 
or expand was in existence at the time of your application. The application was not 
funded. Which of the following occurred? Select one. 

Priority = all 

Funding= 

Non-Funded 


Response Options 

Go to 

1 

The [predominant] course of study was eliminated. 

Question 20 

2 

PDP funding from a different grant was used to support or expand the 
[predominant] course of study. 

Question 5 

3 

External sources of funding were used to support or expand the 
[predominant] course of study. The amount of external funding was equal 
to or greater than the amount requested for the course of study in our PDP 
grant application. 

Question 5 

4 

External sources of funding were used to develop the [predominant] 
course of study. The amount of external funding was smaller than the 
amount requested for that course of study in our PDP grant application 

Question 4 

5 

The [predominant] course of study was maintained with no external 
funding. 

Question 4 
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Question 4 


Text 


Condition 

If the amount of external funding for the [predominant] course of study was smaller 
than the amount requested in your PDP grant application or you received no external 
funding, which of the following steps were implemented? Check all that apply. 

Priority = all 

Funding= 

Non-Funded 


Response Options 

Go to 

a 

Candidates received less financial support than proposed. 


b 

Fewer candidates than proposed were admitted to the [predominant] 
course of study. 


c 

The [predominant] course of study included fewer classes than proposed. 

Question 5 

d 

Other, specify 


e 

None of the above. The [predominant] course of study was implemented 
completely as planned, despite the smaller amount or complete lack of 
external funding. 
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Question 5 


Text 

Condition 

The configuration of your [predominant] course of study may have changed since the 
time of the application. Does the description above accurately reflect the course of 

Priority = all 

study configuration as of the 2008-2009 academic year (summer 2008 to summer 
2009, or a relevant 1 2-month period approximating that time period)? Select one. 

Funding = all 

Project Title: [Pre-filled] 

Roles for which candidate would be prepared: [Pre-filled] 
Disability area: [Pre-filled] 

Degrees for which candidates would be prepared: [Pre-filled] 
Credentials for which candidates would be prepared: [Pre-filled] 
Setting: [Pre-filled] 



Response Options 

Go to 

1 Yes 

Question 7 

0 No 

Question 6 
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Question 6 


Text 

Condition 

You have indicated that the description below does not accurately reflect the 
[predominant] course of study configuration as of the 2008-2009 academic year 
(summer 2008 to summer 2009, or a relevant 12-month period approximating that 
time period). Please make the necessary corrections. 

Priority = all 
Funding = all 


Response 

Go to 

Project Title: 

Roles for which candidates were being prepared: 
Disability area: 

Degree(s) for which candidates were being prepared: 
Credential! s) for which candidates were being prepared: 
Setting (e.g., on a particular campus, online): 

Question 7 
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Question 7 


Text 


Condition 

What were the criteria used to admit candidates into the [predominant] course of 

Priority = all 

study during the 2008-2009 academic year? Check all that apply. 

Funding = all 


Response Options 

Go to 

a 

GPA. Specify minimum GPA, if applicable: 


b 

Past experience related to professional program 


c 

Results of interview 


d 

Review of preadmission portfolio 


e 

Review of recommendation/reference letters 


f 

Review of writing sample 


g 

Statement of candidate’s professional goals 


h 

Prerequisite courses or fieldwork 

Question 8, if 
Combined 

i 

ACT score 

Priority 

j 

SAT score 

Question 12, if 

k 

GRE score 

Leadership 



Priority 

1 

PRAXIS I/Pre-Professional Skills Test (PPST) reading score. Specify 
minimum score, if applicable: 


m 

PRAXIS I/PPST math scores. Specify minimum score, if applicable: 


n 

PRAXIS I/PPST writing scores. Specify minimum score, if applicable: 


0 

Other tests, specify test: . Specify minimum score, 

if applicable: 


P 

Other, specify 
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Question 8 


Text 

Condition 

Did the [predominant] course of study as configured in the 2008-2009 year include 
field-based training, clinical practice, or practicum? For a course of study that takes 
more than one year to complete, consider all years. Select one. 

Priority = 
Combined 

Funding = all 


Response Options 

Go to 

1 Yes 

Question 9 

0 No 

Question 12 


Question 9 

Text 

Condition 

Flow many total hours were candidates expected to spend in field sites, such as 
schools or clinics, in order to complete the entire [predominant] course of study (as it 
was configured in the 2008-2009 year)? For a course of study that takes more than one 
year to complete, include all field work required for completion. Please provide actual 
hours, not credit hours. 

Priority = 
Combined 

Funding = all 


Response 

Go to 

hours 

Question 10 
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Question 10 


Text 


Condition 

Which configuration best describes how field-based training, clinical practice, or 
practicum were included within the [predominant] course of study (as it was 
configured in 2008-2009)? For a course of study that takes more than one year to 
complete, consider field work activities across the entire course of study. Select one. 

Priority = 
Combined 

Funding = all 


Response Options 

Go to 

1 

Field work was performed through separate courses focused on field work 
experiences. 


2 

Field work was embedded within all or some courses throughout the 
[predominant] course of study. 

Question 1 1 

3 

Field work was a combination of separate field work courses and courses 
that have field work embedded within them. 


4 

Other, specify: 
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Question 11 


Text 


Condition 

Approximately what percentages of all field-based training, clinical practice, or 
practicum required to complete the entire [predominant] course of study (as it was 
configured in 2008-2009) were supervised by faculty or staff in each of the following 
categories? For a course of study that takes more than one year to complete, consider 
supervision of all fieldwork required for completion. 

Priority = 
Combined 

Funding = all 


Response Options' 

Go to 

a 

% full-time tenured or tenure-track faculty 


b 

% full-time non-tenured and non-tenure-track faculty 


c 

% full-time staff 


d 

e 

% part-time tenured or tenure-track faculty 
% part-time non-tenured and non-tenure-track faculty 

Question 12 

f 

% part-time staff 


g 

% other 


DK 

Don’t know 



Question 12 

Text 

Condition 

Including hours for all required coursework, theses, and field work, what was the 
minimum number of credit hours required to complete the entire [predominant] course 
of study (as it was configured in 2008-2009)? For a course of study that takes more 
than one year to complete, include hours required for completion. 

Priority = all 
Funding = all 


Response 

Go to 

credit hours 

Question 13 
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Question 13 


Text 


Condition 

How were credit hours organized at your institution during the 2008-2009 academic 
year? Select one. 

Priority = all 
Funding = all 


Response Options 


Go to 

1 Semester hours (16 weeks) 

2 Quarter hours (10 weeks) 

3 Trimester hours ( weeks) 

4 Other credit hours (describe: 

) 

Question 14 


Question 14 


Text 


Condition 

On a 4-point scale, what was the minimum GPA required for retention in and 
completion of the [predominant] course of study during the 2008-2009 
academic year? 

Priority = all 
Funding = all 


Response Options 

Go to 


Minimum GPA for retention and completion 

Question 15, if 

NA 

We had a minimum GPA requirement, but it was not based on 
a 4-point scale. 

Funded 

NA2 

We had a minimum standard for academic performance, but it 
was not based on a numeric average. 

Question 16, ifNon- 
funded 
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Question 15 


Text 


Condition 

What monetary support, if any, did candidates enrolled in the [predominant] 
course of study receive during the 2008-2009 academic year (summer 2008 to 
summer 2009, or a relevant 12-month period approximating that time period) 
specifically from the grant referenced at the beginning of this survey? Please 
enter whole numbers without commas or decimals. 

Priority = all 
Funding = Funded 


Responses 


Go to 

a 

Number of candidates receiving monetary support in academic 
year 2008-2009: 



Grant support in academic year 2008-2009 


b 

Candidate average (mean) $ 

Question 16 

c 

Largest amount for a candidate $ 


d 

Smallest amount for a candidate $ 
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Question 16 


Text 


Condition 

Funded: Other than funds from the [Project Title] during the 2008-200 
academic year, what monetary support did candidates in the [predominant] 
course of study receive specifically because of their enrollment in this course of 
study? Enter whole numbers without commas or decimals. 

Priority = all 
Funding = all 

Non-funded: During the 2008-2009 academic year, what monetary support did 
candidates in the [predominant] course of study receive specifically because of 
their enrollment in this course of study? Enter whole numbers without commas 
or decimals. 



Responses 


Go to 

a 

Number of candidates receiving monetary support in academic 
year 2008-2009: 



Monetary support in academic year 2008-2009 


b 

Candidate average (mean) $ 

Question 17 

c 

Largest amount for a candidate $ 


d 

Smallest amount for a candidate $ 
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Question 17 


Text 


Select one button on each row to indicate the extent to which your [predominant] Priority = see 

course of study focused on each specified area in the 2008-2009 academic year. Use individual 

the “N/A or Not at all” column if you had no focus on a specified area or if the items 

specified area was not relevant to your course of study. 

Funding = all 


Responses 


Responses 


Condition 

NA or 

Small Moderate Great 


Priority = 

Not at 

all 

extent extent extent 


Leadership 


a. 

Research Methodology 

Leadership 


b. 

Personnel preparation 

Leadership 


c. 

Special education 
administration or supervision 

Leadership 


d. 

Policy or advocacy 

Leadership 


e. 

Integrated general and special 
education practice 

Combined 


f. 

Highly qualified teacher 
(HQT) requirements of NCLB 
and IDEA 

Combined 


g- 

Cooperation with SEAs, 
LEAs, or Part C lead agencies 

Combined 


h. 

Integrated general and special 
education practice 

Combined 


i. 

Paraprofessional preparation 

Combined 


j- 

State-identified needs for 
highly qualified personnel 

All 


k. 

Use of evidence-based 
practices in service delivery 

All 


1 . 

Meeting certification, license, 
or endorsement requirements 
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Responses 



Go to 

All 

m. 

Specialized needs of children 
with disabilities from diverse 
cultural and linguistic 
backgrounds 


Combined 

n. 

Extended field experiences or 
practices 


All 

0. 

Field-based training in high- 
need communities 


All 

P- 

Support systems for candidates 
to enhance retention and 
success in the program (e.g., 
tutors, mentors) 


Combined 

q- 

Mentoring and induction for 
program graduates 


All 

r. 

Cultural and linguistic 
diversity in recruiting 
candidates 

Question 18, if 
Leadership 

Combined 

s. 

Collecting data on course of 
study quality 


Combined 

t. 

Involvement of individuals 
with disabilities or their 
parents in the IHE’s program 
planning, implementation, and 
evaluation 


Combined 

u. 

Involvement of parents in their 
children’s educational 
planning and service delivery 

Question 18, if 
Combined 
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Question 18 


Text 

Condition 

Please list all courses within the [predominant] course of study that are new, 
significantly modified, or discontinued since the time of your PDP grant application. 
Please indicate the number of credit hours of the course and note if the course is 
Undergraduate (U), Graduate (G), or Doctoral (D). Please briefly describe the change 
that has occurred to the given course. 

Priority = all 
Funding = all 


Responses 

Go to 

Please check this box □ if there are no new, significantly modified or discontinued 
courses. 

Question 19 

Course Course Title Course No. of Brief description of 

Code Level Credit Changes Made to Course 

Hours 



2 . 

3 . 

4 . 

5 . 

6 . 
7 . 


Question 19 


9 . 

10 . 
11 . 
12 . 

13 . 

14 . 

15 . 

16 . 
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Question 19 


Text 

Condition 

In the table below, describe any changes that have occurred to other components 
related to the [predominant] course of study since the time of your PDP grant 

Priority = all 

application. Changed components might include training units or modules (including 
field experience), recruitment plans, mentoring programs, faculty positions, or the 
overall organization of a course of study. The changes would include the addition, 
significant modification, or discontinuation of any such components. Do not list 
course changes previously listed in this survey. 

Funding = all 


Responses 

Go to 

Please check this box □ if no other changes have occurred. 

Component Brief description of Change Made to 

Component 

1 . 

2. 

Question 20 

3. 

4. 

5. 

6. 

Question 20 
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Question 20 


Text 


Please provide enrollment and completion data for the [predominant] course of study 
for each of the 12-month academic years listed below. 

Note that candidates may be counted more than once across the columns and down the 
rows. 

Check the box below the year if the course of study did not exist for that particular 
year. 





Responses 


Go to 

Condition 

Respondent 

Year=2006 




2005 - 

2006 - 

2007 - 

2008 - 


2006 

2007 

2008 

2009 

Course of study did not exist 

□ 

□ 

□ 

□ 


Priority = all 
Funding = all 


See below for 
other 

conditions. 


Candidates enrolled in the course of study during the academic year: 


Question 21, if 
Combined 


Number of full-time candidates 


Number of part-time candidates 

Number of candidates new to the 

course of study during the academic 

year 

Number of candidates with prior 

general education certification 

Candidates earning degrees or credentials in the course of study 
during the academic year: 


Question 22, if 
Leadership 
and Funded 


Question 23, if 
Leadership 
and Non- 
funded 


Number of candidates who earned 
offers the [pre-filled] degree 

degree 


Number of candidates who earned 
offers the [pre-filled] credential 

credential 
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Responses 

Go to 

Number of candidates who earned 
another degree or credential. 
Specify: 


Number of candidates who earned 
another degree or credential. 
Specify: 


Number of candidates who earned 
another degree or credential. 
Specify: 
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Question 21 


Text 

Condition 

Please list in the table below any standardized exams that candidates were required 
to take in order to demonstrate knowledge and skills for completion of the 
[predominant] course of study. For all PRAXIS II exams, be sure to list the specific 
subject (e.g., PRAXIS II Special Education). 

Priority = 
Combined 

Funding = all 

Check the NA box if a particular exam was not relevant for a particular year. 



Responses 

Go to 

Please check this box □ if the course of study did not require a standardized exam. 

Question 22, if 
Funded 


Question 23, if 
Non-funded 


Condition 

Respondent 
Y ear=2006 




Name of exam or 

2005-2006 

2006- 

2007- 

2008- 

measure 


2007 

2008 

2009 

Name of test 1: 

□ NA 

□ NA 

□ NA 

□ NA 


Number tested 
Number passed 
Median score 

Name of test 2: 


Question 22, if 
Funded 


Question 23, if 

□ NA □ NA □ NA □ NA Non-funded 


Number tested 
Number passed 
Median score 

Name of test 3: 


□ NA □ NA □ NA □ NA 
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Responses 


Go to 


Number tested 
Number passed 
Median score 

Name of test 4: 


□ NA □ NA □ NA □ NA 


Number tested 
Number passed 
Median score 
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Question 22 


Text 

Condition 

Please estimate how funds from the PDP grant referenced at the beginning of 
this survey were spent for your [predominant] course of study in the 2008-2009 
academic year. Please enter whole numbers without commas or decimals. 

Priority = all 
Funding = Funded 


Responses 

Go to 


a 

Monetary support for candidates. (Note: Box is pre-filled 
based on your response to question 15.) [pre-filled] 


b 

Funding for faculty: $ 


c 

Development of curriculum or curriculum elements: $ 


d 

Other. Specify: 



$ 

Question 23 

e 

Other. Specify: 
$ 


f 

Other. Specify: 
$ 


g 

Other. Specify: 
$ 



Question 23 


Text 

Condition 


Do you have a formal data collection to collect information from or about all Priority = all 

candidates who have completed the [predominant] course of study? Select one. 

Funding = all 


Response Options 


1 

Yes 

Question 24 

0 

No 

Question 26 
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Question 24 


Text 

Condition 

How do you collect the data from or about your completers? Check all that apply. 

Priority = all 


Funding = all 


Response Options 

Go to 


a Paper survey of completers 

b Web survey of completers 

c Telephone survey of completers 

d Structured interviews with completers 

Question 25 

e Employer or supervisor survey 

f Structured interviews with supervisors 

g School district or state reports 

h Other, specify 
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Question 25 


Text 

Condition 

Which of the following kinds of information do you collect from or about your 
program completers? Check all that apply. 

Priority = all 
Funding = all 


Response Options 

Go to 


Condition 

a 

Usefulness of training 



b 

Satisfaction with training program 



c 

Employment status (employed/not employed) 



d 

Employment position 



e 

State or district where employed 



f 

Job performance 



g 

Certification status 

Question 26 


h 

Ages or grades of children served 



i 

Primary special education or related services area 



j 

Publications or presentations 



k 

Professional honors 


Combined 

1 

Outcome data on the students taught by the graduates 


Combined 

m 

School administrator’s satisfaction with graduates’ preparation 



n 

Other, specify 
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Question 26 


Text 

Condition 

Please provide dropout data for the [predominant] course of study in academic 
year 2008-2009. Using the reasons provided, indicate the number of candidates 
that left the course of study in academic year 2008-2009 without completing it. 
Make the choice that represents the candidates’ main reason for leaving. Do not 
include a candidate in more than one row. Check all that apply. 

Priority = all 

Funding = Funded 
OR Non-funded and 
not eliminated (q3) 


Responses 

Go to 

Please check this box □ if no candidates left the course of study in academic 
year 2008-2009 without completing it. 

End, if single COS 

Question 27, if 
multiple COSs and 
Funded 

a Financial support for the candidate was terminated 

Question 28, if 
multiple COSs and 
Non-funded 

b Personal reasons 

End, if single COS 

c Poor academic or field-based performance 

Question 27, if 

d Transferred to another training program in special education or 

related services 

multiple COSs and 
Funded 

e Transferred to a training program outside of special education or 

related services 

Question 28, if 

f Other, specify 

multiple COSs and 
Non-funded 

g Reason unknown 
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Question 27 


Text 





Condition 

What monetary support, if any, did candidates enrolled in the other course(s) of study 
receive during the 2008-2009 academic year specifically from the grant referenced at 
the beginning of this survey? 

Priority = all 

Funding = 
Funded 






Number of 
COSs > 1 


Responses 




Go to 



Second 
Course 
of Study 

Third 
Course 
of Study 

Fourth 
Course 
of Study 


a 

Number of candidates receiving monetary 
support 











Monetary support in academic year 2008-2009 



Question 28 

b 

Candidate average (mean) 

$ 

$ 

$ 


c 

Largest amount for a candidate 

$ 

$ 

$ 


d 

Smallest amount for a candidate 

$ 

$ 

$ 



D-27 


Question 28 


Text 

Condition 

Please provide enrollment and completion data for the second course of study (as 
described in question 27) for each of the 12-month academic years listed below. 

Note that candidates may be counted more than once across the columns and down the 
rows. 

Check the box below the year if the course of study did not exist for that particular 
year. 

Priority = all 

Funding = all 

Number of 
COSs > 1 

See below for 
other 

conditions. 


Responses 

Go to 


Condition 


Respondent 

Year=2006 



2005- 

2006- 

2007- 

2008- 


2006 

2007 

2008 

2009 

Course of study did not exist 

□ 

□ 

□ 

□ 

Candidates enrolled in the course of study during the academic 

year: 


Number of full-time candidates 

Number of part-time candidates 

Number of candidates new to the 
course of study during the 
academic year 

Number of candidates with prior 
general education certification 


End, if two 
COSs 


Question 29, if 
three or four 
COSs 


Candidates earning degrees or credentials in the course of study 
during the academic year: 

COS Number of candidates who earned 

offers the [pre-filled] degree 

degree 

COS Number of candidates who earned 

offers the [pre-filled] credential 

credential 
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Responses 

Go to 

Number of candidates who earned 
another degree or credential. 
Specify: 


Number of candidates who earned 
another degree or credential. 
Specify: 


Number of candidates who earned 
another degree or credential. 
Specify: 
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Question 29 


Text 

Condition 

Please provide enrollment and completion data for the third course of study (as 
described in question 27) for each of the 12-month academic years listed below. 

Note that candidates may be counted more than once across the columns and down the 
rows. 

Check the box below the year if the course of study did not exist for that particular 
year. 

Priority = all 

Funding = all 

Number of 
COSs > 2 

See below for 
other 

conditions. 


Responses 

Go to 


Respondent 

Year=2006 

Condition 



2005- 

2006- 

2007- 

2008- 


2006 

2007 

2008 

2009 

Course of study did not exist 

□ 

□ 

□ 

□ 


Candidates enrolled in the course of study during the academic year: 

Number of full-time candidates 


Number of part-time candidates 

Number of candidates new to the 
course of study during the 
academic year 

Number of candidates with prior 
general education certification 

Candidates earning degrees or credentials in the course of study 
during the academic year: 

COS Number of candidates who earned 

offers the [pre-filled] degree 

degree 

COS Number of candidates who earned 

offers the [pre-filled] credential 

credential 


End, if three 
COSs 


Question 30, if 
four COSs 
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Responses 

Go to 

Number of candidates who earned 
another degree or credential. 
Specify: 


Number of candidates who earned 
another degree or credential. 
Specify: 


Number of candidates who earned 
another degree or credential. 
Specify: 
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Question 30 


Text 

Condition 

Please provide enrollment and completion data for the fourth course of study (as 
described in question 27) for each of the 12-month academic years listed below. 

Note that candidates may be counted more than once across the columns and down the 
rows. 

Check the box below the year if the course of study did not exist for that particular 
year. 

Priority = all 

Funding = all 

Number of 
COSs = 4 

See below for 
other 

conditions. 


Responses 

Go to 


Respondent 

Year=2006 

Condition 



2005- 

2006- 

2007- 

2008- 


2006 

2007 

2008 

2009 

Course of study did not exist 

□ 

□ 

□ 

□ 


Candidates enrolled in the course of study during the academic year: 

Number of full-time candidates 


Number of part-time candidates 

Number of candidates new to the 

course of study during the 

academic year 

Number of candidates with prior 

general education certification 

Candidates earning degrees or credentials in the course of study 
during the academic year: 

COS Number of candidates who earned 

offers the [pre-filled] degree 

degree 

COS Number of candidates who earned 

offers the [pre-filled] credential 

credential 
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Responses 

Go to 

Number of candidates who earned 
another degree or credential. 
Specify: 


Number of candidates who earned 
another degree or credential. 
Specify: 


Number of candidates who earned 
another degree or credential. 
Specify: 


Page of questions not answered 

Text 

Condition 

Thank you for your responses. However, you did not complete the questions (s) 
listed below. We appreciate your attention to these items. To review and 
provide a response to any of these questions at this time, please click on the 
relevant question in the Question Guide on the left or on the hotlinks below. 
Please remember that any changes you make will be saved ONLY if you click 
on the “Save & Continue” button below the question you changed. 

If any question is 
left blank that 
should have been 
answered or any 
question left 
incomplete 


Questions 

Go to 

[Pre-filled with incomplete question(s)] 

Comment 

Comment page 

Text 

Condition 

Lastly, please use this space to provide any additional comments about this 
survey and/or the Personnel Development Program. 

Priority = all 

Funding = all 


Response 


Go to 


Submit 
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Submit page 


Text 

Condition 

You have completed the Personnel Development Program (PDP) Higher 
Education Survey, but your data have not yet been submitted. By clicking the 
SUBMIT SURVEY button, your data will be finalized. 

Priority = all 
Funding = all 

Please note: once you click SUBMIT SURVEY, you will not be able to 
review or change your responses, or print a copy of your survey. If you wish 
at this time to review and change any of your responses, please click on the 
desired question to the left. If you revise any of your answers, be sure to click 
the "Save & Continue" button below the question in order to save your changes. 


Before you submit your survey, you may wish to save and print a copy for your 
records: 


Your Completed Personnel Development Prosram (PDP) Higher Education 
Survev 
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Appendix E 


PDP Training Grants Expert Panel Review Rubrics 


Rubric for Courses or Classes 

Background: Linking to PDP Priorities 

The priorities for which these grants were written were: 84.325K (Combined Priority, 2006 and 2007), 
84.325D (Preparation of Leadership Personnel, 2006 and 2007), and 84.325T (Special Education 
Preservice Training Improvement Grants, 2007). This rubric is based on requirements, specifications, and 
purposes of the 2007 priorities, referred to here as K, D, and T. Some key elements of those priorities are 
highlighted here. 

• The ultimate purpose of all competitions is to improve services for children with disabilities. 

• Along with addressing state needs for highly qualified personnel, one of the two purposes of K, 
D, and T grants is to “ensure that those personnel have the skills and knowledge - derived from 
practices that have been determined through research and experience to be successful - that are 
needed to serve those children.” 

• T and K each specify that the proposed program provides “support systems (including tutors, 
mentors, and other innovative practices), to enhance retention and success.” 

• K, D, and T each require “a clear and effective plan for evaluating the extent to which graduates 
of the training program have the knowledge and competencies necessary to provide research- 
based instruction and services that result in improved outcomes for children with disabilities.” 

• K, D, and T all include a requirement that the course of study focus on general education, state 
learning standards, or natural environments. 

• K, D, and T all include a requirement that the course of study “prepare personnel to address the 
specialized needs of children with disabilities from diverse cultural and language backgrounds, 
including limited English proficient children with disabilities” by identifying the skills or 
competencies that personnel need to work effectively with these groups and preparing the 
personnel to use the skills. 


Relevant materials requested for documentation 

• Syllabus of the course 

• CV of the instructor of this course 

• Sample course materials, e.g., presentations, handouts, assignments 

• Assessments of the candidates in this course, including information about how assessments are 
scored and how assessments are linked to course objectives 
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Indicators and Scoring Rubric for New or Significantly Modified Courses or Classes 


Dimension 1: New or Significantly Modified Courses or Classes Deemed of High Quality 


Indicators 

Definitions for Indicators 

1A. The knowledge and skills that are 
emphasized in the course or class 
have a basis in rigorous research. 

The knowledge and skills emphasized in the course or 
class are consistent with the most rigorous and relevant 
scientific evidence to be found in academic journals. This 
indicator focuses only on the knowledge and skills that 
the class or course plans to teach, per the syllabus. 

IB. The knowledge and skills that are 
emphasized in the course or class 
have a basis in successful 
professional experience. 

The knowledge and skills emphasized in the course or 
class are consistent with the successful professional 
experience of practitioners in the field for which 
candidates are being prepared. This indicator focuses 
only on the knowledge and skills that the class or course 
plans to teach, per the syllabus. 

1C. The course or class includes support 
systems to enhance candidate 
retention and success in the course 
or class. 

The course or class uses support systems (such as tutors, 
small class size, faculty support, social media, or other 
innovative practices) to enhance retention and success in 
the course or class. 

ID. Course or class assessments are valid 
and comprehensive measures of 
candidate mastery of the knowledge 
and skills related to course 
objectives. 

Course or class assessments provide valid and 
comprehensive measures of the extent to which course 
completers have the knowledge and skills needed to meet 
course objectives. 

IE. The course or class incorporates 
pedagogy appropriate for the 
knowledge and skills being taught. 

When applicable and appropriate, the course or class 

incoiporates pedagogical approaches that: 

• motivate the user to change his or her practices, 

• reinforce the most important knowledge and skills 
being taught, 

• aid in retention of newly acquired knowledge and 
skills, 

• facilitate transference of newly acquired knowledge 
and skills to other situations, 

• and encourage the use of reflective practice. 

IF. The instructor of the course or class 
is qualified to teach it. 

The instructor’s CV documents knowledge of course or 
class content through education, professional experience, 
practice, and/or research. 
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Criteria for Scoring 

Directions: Circle the number that best describes the quality of the course. 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

1A. The knowledge 

All of the 


Most of the 


None of the 

and skills that are 

knowledge and 


knowledge and 


knowledge or 

emphasized in the 

skills emphasized 


skills 


skills emphasized 

course or class 

in the course or 


emphasized in 


in the course or 

have a basis in 

class are 


the course or 


class are 

rigorous research. 

consistent with 


class are 


consistent with 


the most rigorous 


consistent with 


the most rigorous 


and relevant 


the most rigorous 


and relevant 


scientific evidence 


and relevant 


scientific 


to be found in 


scientific 


evidence to be 


academic 


evidence to be 


found in 


journals. 


found in 


academic 




academic 


journals. 




journals. 




Indicator-Specific Comments: 



Very High 

High 

Moderate 

Low 

Very Low 


Quality 

Quality 

Quality 

Quality 

Quality 

Indicator 

5 

4 

3 

2 

1 

IB. The knowledge 

All of the 


Most of the 


None of the 

and skills that are 

knowledge and 


knowledge and 


knowledge or 

emphasized in the 

skills emphasized 


skills 


skills emphasized 

course or class 

in the course or 


emphasized in 


in the course or 

have a basis in 

class are 


the course or 


class are 

successful 

consistent with 


class are 


consistent with 

professional 

the successful 


consistent with 


the success till 

experience. 

professional 


the successful 


professional 


experience of 


professional 


experience of 


practitioners in 


experience of 


practitioners in 


the field for which 


practitioners in 


the field for 


candidates are 


the field for 


which candidates 


being prepared. 


which candidates 


are being 




are being 
prepared. 


prepared. 

Indicator-Specific Comments: 
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Directions: Circle the number that best describes the quality of the course. 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

1C. The course or 
class includes 
support systems 
to enhance 
candidate 
retention and 
success in the 
course or class. 

The course or 
class includes 
extensive use of 
support systems, 
such as tutors, 
small class size, 
faculty support, 
social media, or 
other innovative 
practices to 
enhance candidate 
retention and 
success in the 
course or class. 


The course or class 
includes some use 
of support systems, 
such as tutors, 
small class size, 
faculty support, 
social media, or 
other innovative 
practices to 
enhance candidate 
retention and 
success in the 
course or class. 


The course or 
class includes 
little if any use of 
support systems 
such as tutors, 
small class size, 
faculty support, 
social media, or 
other innovative 
practices to 
enhance candidate 
retention and 
success in the 
course or class. 


Indicator-Specific Comments: 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

ID. Course or class 
assessments are 
valid and 
comprehensive 
measures of 
candidate 
mastery of the 
knowledge and 
skills related to 
course 
objectives. 

Course or class 
assessments are 
valid and 
comprehensive 
measures of 
candidate mastery 
of the knowledge 
and skills needed 
to meet course 
objectives. 


Course or class 

assessments 

provide 

considerable, valid 
information on 
candidate mastery 
of the knowledge 
and skills needed to 
meet course 
objectives. 


Course or class 
assessments 
provide little or no 
information on 
candidate mastery 
of the knowledge 
and skills needed 
to meet course 
objectives. 

Indicator-Specific Comments: 
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Directions: Circle the number that best describes the quality of the course. 




High 


Low 

Very Low 


Very High Quality 

Quality 

Moderate Quality 

Quality 

Quality 

Indicator 

5 

4 

3 

2 

1 

IE. The course or 

The course or class 


The course or class 


The course or 

class 

incorporates 


incorporates some 


class incorporates 

incorporates 

numerous 


pedagogical 


few if any 

pedagogy 

pedagogical 


approaches 


pedagogical 

appropriate for 

approaches designed 


designed to 


approaches 

the knowledge 

to motivate the 


motivate the 


designed to 

and skills 

learner, reinforce 


learner, reinforce 


motivate the 

being taught. 

course content, aid 


course content, aid 


learner, reinforce 


in retention and 


in retention and 


course content, aid 


transfer of 


transfer of 


in retention and 


knowledge and 


knowledge and 


transfer of 


skills, and 


skills, and 


knowledge and 


encourage 


encourage 


skills, and 


reflection. 


reflection. 


encourage 

reflection. 

Indicator-Specific Comments: 



Very High Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

IF. The instructor 
of the course 
or class is 
qualified to 
teach it. 

The instructor has 
expert knowledge of 
course or class 
content, as 
documented through 
education, 
professional 
practice, experience, 
and/or research. 


The instructor has 
considerable 
knowledge of 
course or class 
content, as 
documented 
through education, 
professional 
practice, 

experience, and/or 
research. 


The instructor has 
little or no 
knowledge of 
course or class 
content, as 
documented 
through education, 
professional 
practice, 
experience, or 
research. 

Indicator-Specific Comments: 


E-5 


Dimension 2: New or Significantly Modified Courses or Classes Deemed of High Relevance/ 
Usefulness 


Indicators 

Definitions for Indicators 

2A. The course or class focuses on 
knowledge and skills that are 
important for supporting the 
inclusion of infants, toddlers, 
children, or youth with disabilities. 

The course or class teaches skills that personnel need to 
directly or indirectly support the inclusion of children 
with disabilities. The course may emphasize knowledge 
and skills important for serving children with disabilities 
in an inclusive setting or natural environment, for 
providing access to the general curriculum, or for 
supporting achievement of state learning standards. 

2B. The course or class prepares 

personnel to address the specialized 
needs of children with disabilities 
from diverse cultural and language 
backgrounds. 

The course or class teaches skills that personnel need to 
directly or indirectly serve children with disabilities from 
diverse cultural and language backgrounds. For example, 
the course may provide opportunities to practice skills in 
a diverse environment, or assessments may test mastery 
of knowledge or skills for effectively serving children 
from diverse cultural and language backgrounds. 

2C. The course or class content is highly 
relevant to the teaching (or other) 
situations for which candidates are 
being prepared. 

The content covered in the course or class is highly 
relevant to the teaching (or other) assignments candidates 
are likely to have and the environments in which they are 
likely to work. 
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Criteria for Scoring 

Directions: Circle the number that best describes the relevance/usefulness of the course. 



Very High 

High 

Moderate 

Low 

Very Low 


Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Indicator 

5 

4 

3 

2 

1 

2A. The course or 

The course or 


The course or class 


The course or 

class focuses on 

class has a 


has a moderate 


class has no 

knowledge and 

strong focus on 


focus on the 


focus on 

skills that are 

the knowledge 


knowledge and 


knowledge and 

important for 

and skills that 


skills that are 


skills that are 

supporting the 

are important 


important for 


important for 

inclusion of 

for supporting 


supporting the 


supporting the 

infants, 

the inclusion of 


inclusion of 


inclusion of 

toddlers, 

children with 


children with 


children with 

children, or 
youth with 
disabilities. 

disabilities. 


disabilities. 


disabilities. 

Indicator-Specific Comments: 



Very High 

High 

Moderate 

Low 

Very Low 


Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Indicator 

5 

4 

3 

2 

1 

2B. The course or 

The course or 


The course or class 


The course or 

class prepares 

class has a 


has a moderate 


class has no focus 

personnel to 

strong focus on 


focus on the skills 


on the skills 

address the 

the skills 


needed to 


needed to work 

specialized 

needed to 


effectively serve 


effectively with 

needs of 

effectively 


children with 


children with 

children with 

serve children 


disabilities from 


disabilities from 

disabilities from 

with disabilities 


diverse cultural 


diverse cultural 

diverse cultural 

from diverse 


and linguistic 


and linguistic 

and language 

cultural and 


backgrounds. 


backgrounds. 

backgrounds. 

linguistic 

backgrounds. 





Indicator-Specific Comments: 


E-7 


Directions: Circle the number that best describes the relevance/usefulness of the course. 



Very High 

High 

Moderate 

Low 

Very Low 


Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Indicator 

5 

4 

3 

2 

1 

2C. The course 

Course or class 


Course or class 


The course or 

or class 

content is highly 


content is highly 


class content is 

content is 

relevant to the 


relevant to the 


highly relevant 

highly 

teaching (or 


teaching (or 


to the teaching 

relevant to 

other) 


other) 


(or other) 

the teaching 

assignments most 


assignments some 


assignments 

(or other) 

candidates are 


candidates are 


few or none of 

situations 

likely to have and 


likely to have and 


the candidates 

for which 

environments in 


environments in 


are likely to 

candidates 

which most are 


which some are 


have or 

are being 

likely to work. 


likely to work. 


environments in 

prepared. 





which they are 
likely to work. 


Indicator-Specific Comments: 


General Comments: 


E-8 


Rubric for Training Units or Modules 


Background: Linking to PDP Priorities 

The priorities for which these grants were written were: 84.325K (Combined Priority, 2006 and 2007), 
84.325D (Preparation of Leadership Personnel, 2006 and 2007), and 84.325T (Special Education 
Preservice Training Improvement Grants, 2007). This rubric is based on requirements, specifications, and 
purposes of the 2007 priorities, referred to here as K, D, and T. Some key elements of those priorities are 
highlighted here. 

• The ultimate purpose of all competitions is to improve services for children with disabilities. 

• Along with addressing state needs for highly qualified personnel, one of the two purposes of K, 
D, and T grants is to “ensure that those personnel have the skills and knowledge - derived from 
practices that have been determined through research and experience to be successful - that are 
needed to serve those children.” 

• T and K each specify that the proposed program provides “support systems (including tutors, 
mentors, and other innovative practices), to enhance retention and success.” 

• K, D, and T each require “a clear and effective plan for evaluating the extent to which graduates 
of the training program have the knowledge and competencies necessary to provide research- 
based instruction and services that result in improved outcomes for children with disabilities.” 

• K, D, and T all include a requirement that the course of study focus on general education, state 
learning standards, or natural environments. 

• K, D, and T all include a requirement that the course of study “prepare personnel to address the 
specialized needs of children with disabilities from diverse cultural and language backgrounds, 
including limited English proficient children with disabilities” by identifying the skills or 
competencies that personnel need to work effectively with these groups and preparing the 
personnel to use the skills. 


Relevant materials requested for documentation 

• A description of the training unit/module 

• CV(s) of key instructors or organizers 

• Sample materials from the training unit/module, e.g., agenda, presentations, handouts 
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Indicators and Scoring Rubric for New or Significantly Modified Training Units or Modules 


Dimension 1: New or Significantly Modified Training Units or Modules Deemed of High Quality 


Indicators 

Definitions for Indicators 

1A. The knowledge and skills 

emphasized in the training unit or 
module have a basis in rigorous 
research. 

The knowledge and skills emphasized in the training unit 
or module are consistent with the most rigorous and 
relevant scientific evidence to be found in academic 
journals. This indicator focuses only on the knowledge 
and skills that the training unit or module plans to teach, 
per the description. 

IB. The knowledge and skills 

emphasized in the training unit or 
module have a basis in successful 
professional experience. 

The knowledge and skills emphasized in the training unit 
or module are consistent with the successful professional 
experience of practitioners in the field for which 
candidates are being prepared. This indicator focuses 
only on the knowledge and skills that the training unit or 
module plans to teach, per the description. 

1C. The training unit or module 

incoiporates pedagogy appropriate 
for the knowledge and skills being 
taught. 

When applicable and appropriate^ the training unit or 

module incorporates pedagogical approaches that: 

• motivate the user to change his or her practices, 

• reinforce the most important knowledge and skills 
being taught, 

• aid in retention of newly acquired knowledge and 
skills, 

• facilitate transference of newly acquired knowledge 
and skills to other situations, 

• and encourage the use of reflective practice. 

ID. The key instructors or organizers are 
knowledgeable of training unit or 
module content. 

The key instructors’ or organizers’ CVs document 
knowledge of training unit or module content through 
education, professional experience, practice, and/or 
research. 
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Criteria for Scoring 

Directions: Circle the number that best describes the quality of the training unit or module. 



Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

1A. The knowledge 
and skills 
emphasized in 
the training unit 
or module have a 
basis in rigorous 
research. 

All of the 
knowledge and 
skills emphasized 
in the training unit 
or module are 
consistent with the 
most rigorous and 
relevant scientific 
evidence to be 
found in academic 
journals. 


Most of the 
knowledge and 
skills emphasized 
in the training unit 
or module are 
consistent with 
the most rigorous 
and relevant 
scientific evidence 
to be found in 
academic 
journals. 


None of the 
knowledge and 
skills emphasized 
in the training unit 
or module are 
consistent with the 
most rigorous and 
relevant scientific 
evidence to be 
found in academic 
journals. 


Indicator-Specific Comments: 



Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

IB. The knowledge 
and skills 
emphasized in 
the training unit 
or module have a 
basis in 
successful 
professional 
experience. 

All of the 
knowledge and 
skills emphasized 
in the training unit 
or module are 
consistent with the 
successful 
professional 
experience of 
practitioners in the 
field for which 
candidates are 
being prepared. 


Most of the 
knowledge and 
skills emphasized 
in the training unit 
or module are 
consistent with 
the successful 
professional 
experience of 
practitioners in 
the field for which 
candidates are 
being prepared. 


None of the 
knowledge and 
skills emphasized 
in the training unit 
or module are 
consistent with the 
successful 
professional 
experience of 
practitioners in the 
field for which 
candidates are 
being prepared. 


Indicator-Specific Comments: 


E-l 1 


Directions: Circle the number that best describes the quality of the training unit or module. 



Very High Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low Quality 

Indicator 

5 

4 

3 

2 

1 

1C. The training 
unit or module 
incorporates 
pedagogy 
appropriate for 
the knowledge 
and skills 
being taught. 

The training unit or 
module incorporates 
numerous 
pedagogical 
approaches designed 
to motivate the 
learner, reinforce 
course content, aid in 
retention and transfer 
of knowledge and 
skills, and encourage 
reflection. 


The training unit or 
module 

incorporates some 
pedagogical 
approaches 
designed to 
motivate the 
learner, reinforce 
course content, aid 
in retention and 
transfer of 
knowledge and 
skills, and 
encourage 
reflection. 


The training unit or 
module 

incorporates few if 
any pedagogical 
approaches 
designed to 
motivate the 
learner, reinforce 
course content, aid 
in retention and 
transfer of 
knowledge and 
skills, and 
encourage 
reflection. 


Indicator-Specific Comments: 



Very High Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low Quality 

Indicator 

5 

4 

3 

2 

1 

ID. The key 
instructors or 
organizers are 
knowledgeable 
of training unit 
or module 
content. 

The key instructors 
or organizers have 
expert knowledge of 
training unit or 
module content, as 
documented through 
education, 

professional practice, 
experience, and/or 
research. 


The key instructors 
or organizers have 
considerable 
knowledge of 
training unit or 
module content, as 
documented 
through education, 
professional 
practice, 

experience, and/or 
research. 


The key instructors 
or organizers have 
little or no 
knowledge of 
training unit or 
module content, as 
documented 
through education, 
professional 
practice, 
experience, or 
research. 

Indicator-Specific Comments: 


E-12 


Dimension 2: New or Significantly Modified Training Units or Modules Deemed of High Relevance/ 
Usefulness 


Indicators 

Definitions for Indicators 

2A. The training unit or module focuses 
on knowledge and skills that are 
important for supporting the 
inclusion of infants, toddlers, 
children, or youth with disabilities. 

The training unit or module teaches skills that personnel 
need to directly or indirectly support the inclusion of 
children with disabilities. The training unit or module 
may emphasize knowledge and skills important for 
serving children with disabilities in an inclusive setting or 
natural environment, for providing access to the general 
curriculum, or for supporting achievement of state 
learning standards. 

2B. The training unit or module prepares 
personnel to address the specialized 
needs of children with disabilities 
from diverse cultural and language 
backgrounds. 

The training unit or module teaches skills that personnel 
need to directly or indirectly serve children with 
disabilities from diverse cultural and language 
backgrounds. For example, training unit or module 
content may emphasize skills important for working in a 
diverse environment, or effectively serving children from 
diverse cultural and language backgrounds. 

2C. The training unit or module content 
is highly relevant to the teaching (or 
other) situations for which 
candidates are being prepared. 

The content covered in the training unit or module is 
highly relevant to the teaching (or other) assignments 
candidates are likely to have and the environments in 
which they are likely to work. 
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Criteria for Scoring 

Directions: Circle the number that best describes the relevance/usefulness of the training unit or module 



Very High 

High 

Moderate 

Low 

Very Low 


Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Indicator 

5 

4 

3 

2 

1 

2A. The training unit 

The training 


The training unit or 


The training unit 

or module 

unit or module 


module has a 


or module has no 

focuses on 

has a strong 


moderate focus on 


focus on 

knowledge and 

focus on 


knowledge and 


knowledge and 

skills that are 

knowledge and 


skills that are 


skills that are 

important for 

skills that are 


important for 


important for 

supporting the 

important for 


supporting the 


supporting the 

inclusion of 

supporting the 


inclusion of 


inclusion of 

infants, toddlers, 

inclusion of 


children with 


children with 

children, or youth 

children with 


disabilities. 


disabilities. 

with disabilities. 

disabilities. 





Indicator-Specific Comments: 



Very High 

High 

Moderate 

Low 

Very Low 


Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Indicator 

5 

4 

3 

2 

1 

2B. The training unit 

The training unit 


The training unit or 


The training unit 

or module 

or module has a 


module has a 


or module has no 

prepares 

strong focus on 


moderate focus on 


focus on skills 

personnel to 

skills needed to 


skills needed to 


needed to 

address the 

effectively serve 


effectively serve 


effectively serve 

specialized needs 

children with 


children with 


children with 

of children with 

disabilities from 


disabilities from 


disabilities from 

disabilities from 

diverse cultural 


diverse cultural 


diverse cultural 

diverse cultural 

and linguistic 


and linguistic 


and linguistic 

and language 
backgrounds. 

backgrounds. 


backgrounds. 


backgrounds. 

Indicator-Specific Comments: 


E-14 


Directions: Circle the number that best describes the relevance/usefulness of the training unit or module. 



Very High 

High 

Moderate 

Low 

Very Low 


Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 

Relevance/ 


Usefulness 

Usefulness 

Usefulness 

Usefulness 

Usefulness 

Indicator 

5 

4 

3 

2 

1 

2C. The training 

Training unit or 


Training unit or 


The training unit 

unit or 

module content 


module content is 


or module 

module 

is highly relevant 


highly relevant to 


content is highly 

content is 

to the 


the assignments 


relevant to the 

highly 

assignments 


some candidates 


assignments few 

relevant to 

most candidates 


are likely to have 


or none of the 

the teaching 

are likely to have 


and environments 


candidates are 

(or other) 

and 


in which some 


likely to have or 

situations for 

environments in 


are likely to 


environments in 

which 

which most are 


work. 


which few or 

candidates 

likely to work. 




none are likely to 

are being 
prepared. 





work. 


Indicator-Specific Comments: 


General Comments: 


E-15 


Rubric for New Faculty 


Background: Linking to PDP Priorities 

The priorities for which these grants were written were: 84.325K (Combined Priority, 2006 and 2007), 
84.325D (Preparation of Leadership Personnel, 2006 and 2007), and 84.325T (Special Education 
Preservice Training Improvement Grants, 2007). This rubric is based on requirements, specifications, and 
purposes of the 2007 priorities, referred to here as K, D, and T. Some key elements of those priorities are 
highlighted here. 

• The ultimate purpose of all competitions is to improve services for children with disabilities. 

• Along with addressing state needs for highly qualified personnel, one of the two purposes of K, 
D, and T grants is to “ensure that those personnel have the skills and knowledge - derived from 
practices that have been determined through research and experience to be successful - that are 
needed to serve those children.” 

• K, D, and T each require “a clear and effective plan for evaluating the extent to which graduates 
of the training program have the knowledge and competencies necessary to provide research- 
based instruction and services that result in improved outcomes for children with disabilities.” 

• K, D, and T all include a requirement that the course of study focus on general education, state 
learning standards, or natural environments. 

• K, D, and T all include a requirement that the course of study “prepare personnel to address the 
specialized needs of children with disabilities from diverse cultural and language backgrounds, 
including limited English proficient children with disabilities” by identifying the skills or 
competencies that personnel need to work effectively with these groups and preparing the 
personnel to use the skills. 

• D and K require that the course of study provide “integrated training and practice opportunities”; 
T requires “extended clinical learning opportunities, field experiences, or supervised practica.” 


Relevant materials requested for documentation 

• CV of faculty member, including courses taught in the past 

• Syllabi of all courses to which faculty member was assigned 

• Sample assessments of candidates developed or selected by faculty member, including 
information about how the assessments are scored 
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Indicators and Scoring Rubric for New Faculty 


New Faculty Deemed of High Quality 


Indicators 

Definitions for Indicators 

1A. The new faculty member has the 
necessary background and 
experience to teach the courses to 
which s/he is assigned. 

The new faculty member is prepared to teach the 
knowledge and skills, derived from practices that have 
been determined through research and experience to be 
successful, that are relevant to the courses to which s/he 
is assigned. 

IB. The new faculty member is skillful 
in developing and using valid and 
comprehensive measures of 
candidate mastery of knowledge and 
skills. 

The new faculty member demonstrates skill in 
developing (or selecting) and using valid and 
comprehensive measures of candidate mastery of 
knowledge and skills. 

1C. The new faculty member uses 
pedagogical approaches that are 
appropriate for the knowledge and 
skills being taught. 

The new faculty member uses pedagogical approaches 

that, when applicable and appropriate, can: 

• motivate the user to change his or her practices, 

• reinforce the most important knowledge and skills 
being taught, 

• aid in retention of newly acquired knowledge and 
skills, 

• facilitate transfer of newly acquired knowledge and 
skills to other situations, 

• and encourage the use of reflective practice. 

ID. The new faculty member has the 
necessary background and 
experience to contribute to a focus 
on general education, state learning 
standards, or natural environments. 

The new faculty member demonstrates, through practical 
experience, publications, presentations, and/or courses 
taught in the past, the necessary background and 
experience to contribute to a focus on general education, 
state learning standards, or natural environments. 

IE. The new faculty member has 
knowledge of and experience with 
the specialized needs of children 
with disabilities from diverse cultural 
and linguistic backgrounds. 

The new faculty member demonstrates knowledge of and 
experience with the specialized needs of children with 
disabilities from diverse cultural and linguistic 
backgrounds through practical experience, publications, 
presentations, or courses taught in the past. 

IF. The new faculty member is 

experienced with the teaching (or 
other) situations for which 
candidates are being prepared. 

The new faculty member has professional experience 
with the teaching (or other) assignments candidates are 
likely to have and the environments in which they are 
likely to work. 

1G. The new faculty member has 

expertise in integrated training and 
practice. 

The new faculty member has expertise in the use of 
integrated training and practice such as clinical learning 
opportunities, field experiences, or supervised practica, 
shown through professional experience and research. 
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Criteria for Scoring 

Directions: Circle the number that best describes the quality of the new faculty member. 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

1A. The new faculty 
member has the 
necessary 
background and 
experience to 
teach the courses 
to which s/he is 
assigned. 

The new faculty 
member is 
prepared to teach 
all or nearly all of 
the knowledge 
and skills that are 
relevant to the 
courses to which 
s/he is assigned. 


The new faculty 
member is 
prepared to teach 
some but not all 
of the knowledge 
and skills that are 
relevant to the 
courses to which 
s/he is assigned. 


The new faculty 
member is 
prepared to teach 
few if any of the 
knowledge and 
skills that are 
relevant to the 
courses to which 
s/he is assigned. 


Indicator-Specific Comments: 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

IB. The new faculty 

The new faculty 


The new faculty 


The new faculty 

member is 

member 


member 


member 

skillful in 

demonstrates 


demonstrates 


demonstrates little 

developing and 

exceptional skill 


competence in 


if any skill in 

using valid and 

in developing (or 


developing (or 


developing (or 

comprehensive 

selecting) and 


selecting) and 


selecting) and 

measures of 

using valid and 


using valid and 


using valid and 

candidate 

comprehensive 


comprehensive 


comprehensive 

mastery of 

measures of 


measures of 


measures of 

knowledge and 

candidate mastery 


candidate mastery 


candidate mastery 

skills. 

of knowledge and 


of knowledge and 


of knowledge and 


skills. 


skills. 


skills. 


Indicator-Specific Comments: 
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Directions: Circle the number that best describes the quality of the new faculty member. 


Indicator 

Very High Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

1C. The new faculty 

The new faculty 


The new faculty 


The new faculty 

member uses 

member makes 


member makes 


member makes 

pedagogical 

extensive use of 


some use of 


little if any use of 

approaches that 

pedagogical 


pedagogical 


pedagogical 

are appropriate for 

approaches that are 


approaches that are 


approaches that 

the knowledge 

likely to motivate 


likely to motivate 


are likely to 

and skills being 

the learner, reinforce 


the learner, 


motivate the 

taught. 

course content, aid 


reinforce course 


learner, reinforce 


in retention and 


content, aid in 


course content. 


transfer of 


retention and 


aid in retention 


knowledge and 


transfer of 


and transfer of 


skills, and 


knowledge and 


knowledge and 


encourage 


skills, and 


skills, and 


reflection. 


encourage 


encourage 




reflection. 


reflection. 


Indicator-Specific Comments: 


Indicator 

Very High Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

ID. The new faculty 

The new faculty 


The new faculty 


The new faculty 

member has the 

member has the 


member has the 


member does not 

necessary 

background and 


background and 


have the 

background and 

experience to make 


experience to make 


background or 

experience to 

an exemplary 


an adequate 


experience to 

contribute to a 

contribution to a 


contribution to a 


contribute to a 

focus on general 

focus on general 


focus on general 


focus on general 

education, state 

education, state 


education, state 


education, state 

learning 

learning standards, 


learning standards, 


learning 

standards, or 

or natural 


or natural 


standards, or 

natural 

environments. 


environments. 


natural 

environments. 





environments 


Indicator-Specific Comments: 


E-19 


Directions: Circle the number that best describes the quality of the new faculty member. 



Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

IE. The new faculty 
member has 
knowledge of 
and experience 
with the 

specialized needs 
of children with 
disabilities from 
diverse cultural 
and linguistic 
backgrounds. 

The new faculty 
member has expert 
knowledge of and 
extensive 

experience with the 
specialized needs 
of children with 
disabilities from 
diverse cultural and 
linguistic 
backgrounds. 


The new faculty 
member has 
adequate 

knowledge of and 
limited experience 
with the 

specialized needs 
of children with 
disabilities from 
diverse cultural 
and linguistic 
backgrounds. 


The new faculty 
member has little 
if any knowledge 
of or experience 
with the 

specialized needs 
of children with 
disabilities from 
diverse cultural 
and linguistic 
backgrounds. 


Indicator-Specific Comments: 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

IF. The new faculty 
member is 
experienced with 
the teaching (or 
other) situations 
for which 
candidates are 
being prepared. 

The new faculty 
member has 
extensive 

experience with the 
teaching (or other) 
situations for 
which candidates 
are being prepared. 


The new faculty 
member has 
considerable 
experience with 
the teaching (or 
other) situations 
for which 
candidates are 
being prepared. 


The new faculty 
member has little 
or no experience 
with the teaching 
(or other) 
situations for 
which candidates 
are being prepared. 


Indicator-Specific Comments: 
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Directions: Circle the number that best describes the quality of the new faculty member. 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

1G. The new 
faculty 
member has 
expertise in 
integrated 
training and 
practice. 

The new faculty 
member has 
extensive 
expertise in the 
use of integrated 
training and 
practice such as 
clinical learning 
opportunities, 
field experiences, 
or supervised 
practica. 


The new faculty 
member has 
considerable 
expertise in the use 
of integrated 
training and 
practice such as 
clinical learning 
opportunities, field 
experiences, or 
supervised 
practica. 


The new faculty 
member has little 
or no expertise in 
the use of 
integrated training 
and practice such 
as clinical learning 
opportunities, 
field experiences, 
or supervised 
practica. 


Indicator-Specific Comments: 


General Comments: 
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Rubric for Field Work 


Background: Linking to PDP Priorities 

The priorities for which these grants were written were: 84.325K (Combined Priority, 2006 and 2007), 
84.325D (Preparation of Leadership Personnel, 2006 and 2007), and 84.325T (Special Education 
Preservice Training Improvement Grants, 2007). This rubric is based on requirements, specifications, and 
purposes of the 2007 priorities, referred to here as K, D, and T. Some key elements of those priorities are 
highlighted here. 

• The ultimate purpose of all competitions is to improve services for children with disabilities. 

• Along with addressing state needs for highly qualified personnel, one of the two purposes of K, 
D, and T grants is to “ensure that those personnel have the skills and knowledge - derived from 
practices that have been determined through research and experience to be successful - that are 
needed to serve those children.” 

• T and K each specify that the proposed program provides “support systems (including tutors, 
mentors, and other innovative practices), to enhance retention and success.” 

• K, D, and T each require “a clear and effective plan for evaluating the extent to which graduates 
of the training program have the knowledge and competencies necessary to provide research- 
based instruction and services that result in improved outcomes for children with disabilities.” 

• D and K require that the course of study provide “integrated training and practice opportunities”; 
T requires “extended clinical learning opportunities, field experiences, or supervised practica.” 


Relevant materials requested for documentation 

• Field work plan or syllabus of field work course 

• CV(s) of faculty primarily responsible for leading or supervising field work 

• Field work handbook 

• Instruments used to assess candidates, including sample faculty or supervisor evaluations of 
candidates 

• Materials used to train supervisors 
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Indicators and Scoring Rubric for Field Work 


Field Work Deemed of High Quality 


Indicators 

Definitions for Indicators 

1A. Field work experiences are highly 
relevant to the teaching (or other) 
situations for which candidates are 
being prepared. 

Field work experiences are highly relevant to the teaching 
(or other) assignments candidates are likely to have and 
the environments in which they are likely to work. 

IB. The goals of the field work are 

clearly articulated and focus on issues 
that are relevant for candidate success 
and effectiveness. 

The goals of the field work are clearly articulated and 
focus on relevant issues, such as providing a mechanism 
for modeling and demonstration, promoting candidate 
insight into the curriculum and organization of the school 
and classroom, or providing opportunities for candidate 
analysis, reflection and decision-making. 

1C. The knowledge and skills that are 
emphasized in the field work have a 
basis in rigorous research. 

The knowledge and skills emphasized in the field work 
are consistent with the most rigorous and relevant 
scientific evidence to be found in academic journals. This 
indicator focuses only on the knowledge and skills that the 
field work plans to develop, per the field work description 
or syllabus. 

ID. The knowledge and skills that are 
emphasized in the field work have a 
basis in successful professional 
experience. 

The knowledge and skills emphasized in the field work 
are consistent with the successful professional experience 
of practitioners in the field for which candidates are being 
prepared. This indicator focuses only on the knowledge 
and skills that the field work plans to develop, per the field 
work description or syllabus. 

IE. The field work includes support 
systems beyond the supervisor to 
enhance candidate retention and 
success in the field experience. 

The field work program uses support systems (such as 
peer support, mentors, or other innovative practices) to 
enhance retention and success in the field experience. 

IF. Candidate evaluations are valid and 
comprehensive measures of candidate 
mastery of the skills addressed by the 
field work. 

Candidate evaluations provide valid and comprehensive 
measures of the extent to which field work completers 
have mastered the skills needed to meet field work 
objectives. 

1G. Supervising faculty members are 
experienced with the teaching (or 
other) situations for which candidates 
are being prepared. 

The supervising faculty members (who may be adjunct or 
ad hoc) have the necessary experience with the teaching 
(or other) assignments candidates are likely to have and 
the environments in which they are likely to work. 

1H. Supervising faculty members have 
expertise in integrated training and 
practice. 

The supervising faculty members have expertise in the use 
of integrated training and practice such as clinical learning 
opportunities, field experiences, or supervised practica, 
shown through professional experience and research. 

11. The field work provides for training 
of IHE-based supervisors and field 
supervisors. 

The field work provides for the training of IHE-based and 
field-based supervisors, which may include a handbook, 
Power Point slides, or other written materials; in-person 
training/orientation; or joint planning. 
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Criteria for Scoring 

Directions: Circle the number that best describes the quality of the field work. 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low Quality 

Indicator 

5 

4 

3 

2 

1 

1A. Field work 
experiences are 
highly relevant 
to the teaching 
(or other) 
situations for 
which 

candidates are 
being prepared. 

The field work 
experiences are 
highly relevant 
to the teaching 
(or other) 
assignments 
most candidates 
are likely to have 
and 

environments in 
which most are 
likely to work. 


The field work 
experiences are 
highly relevant to 
the teaching (or 
other) assignments 
some candidates 
are likely to have 
and environments 
in which some are 
likely to work. 


The field work 
experiences are 
highly relevant to 
the teaching (or 
other) assignments 
few or none of the 
candidates are 
likely to have and 
environments in 
which they are 
likely to work. 

Indicator-Specific Comments: 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low Quality 

Indicator 

5 

4 

3 

2 

1 

IB. The goals of 
the field work 
are clearly 
articulated and 
focus on issues 
that are relevant 
for candidate 
success and 
effectiveness. 

The goals of the 
field work are 
clearly 

articulated in the 
plan or syllabus 
and focus on 
issues that are 
relevant for 
candidate 
success and 
effectiveness. 


The goals of the 
field work are 
referenced in the 
plan or syllabus, 
although not 
clearly or precisely 
articulated, and/or 
the goals focus on 
most of the issues 
that are relevant 
for candidate 
success and 
effectiveness. 


The goals of the 
field work are not 
articulated in the 
plan or syllabus, or 
the goals focus on 
few or none of the 
issues that are 
relevant for 
candidate success 
and effectiveness. 


Indicator-Specific Comments: 
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Directions: Circle the number that best describes the quality of the field work. 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low Quality 

5 

4 

3 

2 

1 

1C. The knowledge 
and skills that 
are emphasized 
in the field work 
have a basis in 
rigorous 
research. 

All of the 
knowledge and 
skills emphasized 
in the field work 
are consistent with 
the most rigorous 
and relevant 
scientific evidence 
to be found in 
relevant academic 
journals. 


Most of the 
knowledge and 
skills emphasized 
in the field work 
are consistent with 
the most rigorous 
and relevant 
scientific evidence 
to be found in 
relevant academic 
journals. 


None of the 
knowledge or skills 
emphasized in the 
field work are 
consistent with the 
most rigorous and 
relevant scientific 
evidence to be 
found in relevant 
academic journals. 

Indicator-Specific Comments: 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low Quality 

Indicator 

5 

4 

3 

2 

1 

ID. The knowledge 
and skills that 
are emphasized 
in the field work 
have a basis in 
successful 
professional 
experience. 

All of the 
knowledge and 
skills emphasized 
in the field work 
are consistent with 
the successful 
professional 
experience of 
practitioners in the 
field for which 
candidates are 
being prepared. 


Most of the 
knowledge and 
skills emphasized 
in the field work 
are consistent with 
the successful 
professional 
experience of 
practitioners in the 
field for which 
candidates are 
being prepared. 


None of the 
knowledge or skills 
emphasized in the 
field work are 
consistent with the 
successful 
professional 
experience of 
practitioners in the 
field for which 
candidates are 
being prepared. 


Indicator-Specific Comments: 
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Directions: Circle the number that best describes the quality of the field work. 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

IE. The field work 
includes support 
systems beyond 
the supervisor to 
enhance candidate 
retention and 
success in the 
field experience. 

The field work 
includes extensive 
use of support 
systems, such as 
peer support, 
mentors, or other 
innovative 
practices to 
enhance retention 
and success in the 
field experience. 


The field work 
includes some use 
of support systems, 
such as peer 
support, mentors, 
or other innovative 
practices to 
enhance retention 
and success in the 
field experience. 


The field work 
includes little if 
any use of 
support systems 
to enhance 
candidate 
retention and 
success in the 
field work. 


Indicator-Specific Comments: 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

IF. Candidate 
evaluations are 
valid and 
comprehensive 
measures of 
candidate mastery 
of the skills 
addressed by the 
field work. 

The candidate 
assessments are 
valid and 
comprehensive 
measures of 
candidate mastery 
of the skills 
addressed by the 
field work. 


The candidate 

assessments 

provide 

considerable, valid 
information on 
candidate mastery 
of the skills 
addressed by the 
field work. 


The candidate 
assessments 
provide little or 
no information 
on candidate 
mastery of the 
skills addressed 
by the field 
work. 

Indicator-Specific Comments: 
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Directions: Circle the number that best describes the quality of the field work. 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

1G. Supervising 
faculty 
members are 
experienced 
with the 
teaching (or 
other) situations 
for which 
candidates are 
being prepared. 

The supervising 
faculty members 
have extensive 
experience with 
the teaching (or 
other) situations 
for which 
candidates are 
being prepared. 


The supervising 
faculty members 
have considerable 
experience with 
the teaching (or 
other) situations 
for which 
candidates are 
being prepared. 


The supervising 
faculty members 
have little or no 
experience with 
the teaching (or 
other) situations 
for which 
candidates are 
being prepared. 

Indicator-Specific Comments: 



Very High 
Quality 

High 

Quality 

Moderate 

Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

1H. Supervising 
faculty 

members have 
expertise in 
integrated 
training and 
practice. 

The supervising 
faculty members 
have extensive 
expertise in the 
use of integrated 
training and 
practice such as 
clinical learning 
opportunities, 
field experiences, 
and/or supervised 
practica. 


The supervising 
faculty members 
have considerable 
expertise in the 
use of integrated 
training and 
practice such as 
clinical learning 
opportunities, 
field experiences, 
and/or supervised 
practica. 


The supervising 
faculty members 
have little or no 
expertise in the 
use of integrated 
training and 
practice such as 
clinical learning 
opportunities, 
field experiences, 
or supervised 
practica. 


Indicator-Specific Comments: 
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Directions: Circle the number that best describes the quality of the field work. 



Very High Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 

Quality 

Indicator 

5 

4 

3 

2 

1 

11. The field work 
provides for 
training of IHE- 
based 

supervisors and 
field 

supervisors. 

The field work includes 
extensive and ongoing 
training of field 
supervisors and IHE- 
based supervisors, 
which may include a 
handbook, PowerPoint 
slides, or other written 
materials; in-person 
training/orientation; 
and/or joint planning. 


The field work 
includes some but 
not extensive or 
ongoing training of 
field supervisors or 
IHE-based 
supervisors, which 
may include a 
handbook, 
PowerPoint slides, 
or other written 
materials; in-person 
training/ orientation; 
and/or joint 
planning. 


The field work 
includes little or 
no field 
supervisor or 
IHE-based 
supervisor 
training. 


Indicator-Specific Comments: 


General Comments: 
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Rubric for Mentoring Programs 


Background: Linking to PDP Priorities 

The priorities for which these grants were written were: 84.325K (Combined Priority, 2006 and 2007), 
84.325D (Preparation of Leadership Personnel, 2006 and 2007), and 84.325T (Special Education 
Preservice Training Improvement Grants, 2007). This rubric is based on requirements, specifications, and 
purposes of the 2007 priorities, referred to here as K, D, and T. Some key elements of those priorities are 
highlighted here. 

• The ultimate purpose of all competitions is to improve services for children with disabilities. 

• Along with addressing state needs for highly qualified personnel, one of the two purposes of K, 
D, and T grants is to “ensure that those personnel have the skills and knowledge - derived from 
practices that have been determined through research and experience to be successful - that are 
needed to serve those children.” 

• T and K each specify that the proposed program provides “support systems (including tutors, 
mentors, and other innovative practices), to enhance retention and success.” 


Relevant materials requested for documentation 

• Mentoring plan 

• Program data including number of mentors, number of candidates served, and schedule and 
structure of mentor-mentee interactions 

• Materials used to train mentors 
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Indicators and Scoring Rubric for Mentoring Programs 


Mentoring Program Deemed of High Quality 


Indicators 

Definitions for Indicators 

1A. The focus of the mentoring program 
is highly relevant to the teaching (or 
other) situations for which 
candidates are being prepared. 

The focus of the mentoring program is highly relevant to 
the teaching (or other) assignments candidates are likely 
to have and the environments in which they are likely to 
work 

IB. The mentoring program supports 
extensive contact between the 
mentor and mentee. 

The program supports frequent interactions between the 
mentor and mentee. Interactions may be varied and may 
include scheduled meetings, unscheduled meetings, 
telephone contact, written communication, and/or 
observation. 

1C. The mentoring program provides a 
broad range of support. 

The mentoring program provides various types of 
support, including social/emotional support; information 
about materials, resources, policies, and procedures; 
academic guidance; and/or professional guidance, such as 
mentoring for instruction. 

ID. The mentoring program matches 
mentors with mentees based on 
professional experiences, academic 
interests, and personality match. 

Mentors and mentees are matched based on identified 
criteria, such as professional experiences, academic 
interests, and personality match. 

IE. The goals of the mentoring program 
are clearly articulated and focus on 
issues that are relevant for candidate 
success and effectiveness. 

The goals of the mentoring program are clearly 
articulated and focus on relevant issues, such as 
improving candidate performance, increasing candidate 
retention in the profession, transmitting best-practice 
cultural norms for the population being served, 
facilitating recruitment, and/or promoting the well-being 
and satisfaction of the candidates. 

IF. The mentoring program provides for 
mentor training. 

The mentoring program provides for mentor training, 
which may include a handbook, PowerPoint slides, or 
other written materials; in-person training/orientation; or 
joint planning. Mentor training may include both initial 
and ongoing learning. 

1G. The mentoring program maintains 
mentor-to-candidate ratios that are 
consistent with high quality 
mentoring. 

The mentoring program maintains mentor-to-mentee 
ratios that allow mentors to devote the time, attention, 
and energy required to provide high quality mentoring. 
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Criteria for Scoring 

Directions: Circle the number that best describes the quality of the mentoring program. 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

1A. The focus of the 
mentoring 
program is 
highly relevant 
to the teaching 
(or other) 
situations for 
which candidates 
are being 
prepared. 

The focus of the 
mentoring 
program is 
highly relevant 
to the teaching 
(or other) 
assignments 
most mentees 
are likely to 
have and 
environments in 
which most are 
likely to work. 


The focus of the 
mentoring program 
is highly relevant 
to the teaching (or 
other) assignments 
some mentees are 
likely to have and 
environments in 
which some are 
likely to work. 


The focus of the 
mentoring 
program is highly 
relevant to the 
teaching (or other) 
assignments few 
or none of the 
mentees are likely 
to have or 
environments in 
which they are 
likely to work. 


Indicator-Specific Comments: 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

IB. The mentoring 
program 
supports 

extensive contact 
between the 
mentor and 
mentee. 

The program 
requires and 
supports 
frequent 
interactions 
between the 
mentor and the 
mentee. 


The program 
provides 

opportunities for a 
considerable 
number of 
interactions 
between the mentor 
and the mentee. 


The program 
provides few 
opportunities for 
interactions 
between the 
mentor and the 
mentee. 

Indicator-Specific Comments: 
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Directions: Circle the number that best describes the c 

uality of the mentoring program. 

Indicator 

Very High Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

1C. The mentoring 
program provides 
a broad range of 
support. 

The mentoring 
program provides a 
broad range of 
support, including 
social/emotional 
support; information 
about materials, 
resources, policies, 
and procedures; 
academic guidance; 
and/or professional 
guidance. 


The mentoring 
program provides 
some but not all types 
of support, including 
social/emotional 
support; information 
about materials, 
resources, policies, 
and procedures; 
academic guidance; 
and/or professional 
guidance. 


The 

mentoring 
program 
provides few 
if any specific 
types of 
support. 

Indicator-Specific Comments: 


Indicator 

Very High Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

ID. The mentoring 
program matches 
mentors with 
mentees based on 
professional 
experiences, 
academic 
interests, and 
personality match. 

The program 
matches mentors 
and mentees based 
on multiple 
indicators, including 
professional 
experiences, 
academic interests, 
and personalities. 


The program matches 
mentors and mentees 
based on one of the 
following: 
professional 
experiences, 
academic interests, 
and/or personality 
match. 


The program 
has little if 
any 

substantive 
basis for 
matching 
mentors and 
mentees. 

Indicator-Specific Comments: 
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Directions: Circle the number that best describes the quality of the mentoring program. 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

IE. The goals of the 
mentoring 
program are 
clearly articulated 
and focus on 
issues that are 
relevant for 
candidate success 
and effectiveness. 

The goals of the 
mentoring 
program are 
clearly articulated 
in the plan or 
other materials, 
and focus on all of 
the issues that are 
relevant for 
candidate success 
and effectiveness. 


The goals of the 
mentoring program 
are referenced in the 
plan or other 
materials, although 
not clearly or 
precisely articulated, 
and focus on some 
of the issues that are 
relevant for 
candidate success 
and effectiveness. 


The goals of the 
mentoring 
program are not 
articulated in the 
plan or other 
materials or focus 
on few or none of 
the issues that are 
relevant for 
candidate success 
and effectiveness. 


Indicator-Specific Comments: 



Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

Indicator 

5 

4 

3 

2 

1 

IF. The mentoring 
program provides 
for mentor 
training. 

The mentoring 
program includes 
extensive mentor 
training, which 
may include a 
handbook, 
PowerPoint slides, 
or other written 
materials; in- 
person training/ 
orientation; and/or 
joint planning. 


The mentoring 
program includes 
some but not 
extensive mentor 
training, which may 
include a handbook, 
PowerPoint slides, 
or other written 
materials; in-person 
training/ orientation; 
and/or joint 
planning. 


The mentoring 
program includes 
little or no 
mentor training. 

Indicator-Specific Comments: 
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Directions: Circle the number that best describes the quality of the mentoring program. 


Indicator 

Very High 
Quality 

High 

Quality 

Moderate Quality 

Low 

Quality 

Very Low 
Quality 

5 

4 

3 

2 

1 

1G. The mentoring 
program 
maintains 
mentor-to- 
candidate ratios 
that are 

consistent with 
high quality 
mentoring. 

The mentoring 
program sets 
mentor-to- 
candidate ratios 
consistent with 
mentoring of the 
highest quality. 


The mentoring 
program sets 
mentor-to- 
candidate ratios 
consistent with 
mentoring of 
moderate quality. 


The mentoring 
program does not 
address the ratio 
of mentors to 
candidates or the 
ratios are not 
consistent with 
adequate 
mentoring. 


Indicator-Specific Comments: 


General Comments: 
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Appendix F 


Expert Panel Review Inter-rater Agreement 

This evaluation involved multiple sets of raters assigning ratings to products sampled for 
the Study of PDP National Centers and components sampled for the Study of PDP Training 
Grant Applicants. When multiple raters are assigned in such a way, it is important to consider 
two related but conceptually distinct concepts: inter-rater agreement (IRA) and inter-rater 
reliability (IRR). Levels of IRA index the extent to which raters assign the same scores to 
common targets, while IRR indicates the extent to which raters provide consistency in relative 
judgments. For example, if one rater provides ratings that are 2 points higher than another rater, 
measures of IRR will be high, while IRA will be lower. For the expert panel review of both 
products sampled from centers and components sampled from courses of study , IRA was 
identified as more conceptually appropriate for assessing the extent to which raters provided 
similar ratings. 

There are a variety of methods for assessing IRA and IRR. For example, Cohen’s kappa 
is a classic measure of agreement, but requires additional modifications (e.g., weighting 
schemes) to handle ranked data and is primarily designed around assessing agreement 
dyadically — e.g., between pairs of individual raters on individual items. The various intraclass 
correlation coefficients (c.f., Shrout and Fleiss 1979) are widely used to assess the consistency of 
ratings, but provide measures based on both IRA and IRR, rather than IRA alone (LeBreton and 
Senter 2008), do not provide a measure of agreement for individual products, and are based on 
mean dimension-level ratings, rather than item-level responses. These measures were less than 
optimal for purposes of the current evaluation. 

We determined Rwg (j) (James, Demaree, and Wolf 1984) to fit the specific purposes of 
the expert panel reviews in the current evaluation in that ( 1 ) it focuses on inter-rater agreement 
rather than inter-rater consistency, (2) the ratings are provided by three panel reviewers rather 
than pairs (in which case, for example, kappa may be more appropriate), and (3) all ratings 
would be accepted as long as they fall within a reasonable agreement range (e.g., higher than low 
or medium agreement). The Rwg (j) measure is widely used in the psychometric, management, 
and psychology literatures; is applicable to multi-item ordinal response scales; and provides a 
measure of IRA for each product or component being rated (as opposed to a single measure of 
IRA for the entire study) based on a definite number of raters. Rwg(j) is given by the following 
formula: 



F-l 


where J = the number of items in the rating scale, (s_(x_j) A 2 ) is the mean of the observed 
variances for the J observed items, and a_E A 2 is the expected variance when there is a complete 
lack of agreement among raters (e.g., where raters respond randomly to items). For the current 
report, o_E A 2 =2, based on the expected variance of a uniform response distribution for a 5-item 
response scale. 

RwgO) ranges from 0 to 1, with 0 indicating a complete lack of agreement and 1 
indicating perfect agreement. The analysis was conducted using the rwg.j function from the 
multilevel package in R version 2.10.1 (R Foundation for Statistical Computing 2010); codes are 
also provided for SPSS in LeBreton and Senter (2008). Because the Rwg(j) formula expects 
numeric ratings, “NA” ratings were treated as missing data for purposes of the Rwg(j) analysis. 
Missing data are known to have an effect on Rwg(j) estimates (Newman & Sin 2009), such that 
responses missing not at random tend to produce overestimates of Rwg(j), with the level of bias 
dependent on the level of missingness in the data and the magnitude of the relationship between 
missingness and the variable of interest. As with many metrics of IRA, what constitutes a “high” 
or “low” level of agreement will ultimately depend on the purposes of the ratings, but a review 
by LeBreton & Senter (2008) suggested the following interpretations of Rwg(j): ranges from 
0.00 to 0.30 represent a lack of agreement; 0.31 to 0.50 represents weak agreement; 0.51 to 0.70 
is moderate agreement; 0.71 to 0.90 is strong agreement; and 0.91 to 1.00 is very strong 
agreement. In order to identify products for further reconciliation and re-review by raters, 
products were selected for reconciliation if they had an Rwg(j) of 0.30 or less on either quality or 
relevance (the two dimensions). 

In addition, for each product or component thus identified, raters were provided with a 
list of specific indicators for which the greatest level of disagreement existed, defined by the 
inter-rater variance for specific indicators. These were identified by selecting indicators for 
which ratings had a variance of two or more; this threshold was selected based on the expected 
variance of a uniform distribution of an ordinal scale with five response categories, such that 
indicators with variances less than two tended to have greater levels of agreement than might be 
expected based on chance responding. Note that this threshold does not represent a statistical test 
of inter-rater agreement; rather, it was a heuristic designed to help raters quickly identify and 
resolve the largest discrepancies in their ratings in an efficient manner. 
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Appendix G 

Technical Working Group Members 


Technical Working Group Members for the Design Contract (ED-04-CO-0059/0Q18) 

Robert Algozzine, University of North Carolina-Charlotte 

Mary Brownell, University of Florida 

Mary Compton, University of North Carolina-Greensboro 

Vincent Connelly, University of New Hampshire 

Tom Cook, Northwestern University 

Kay Ferrell, University of Northern Colorado 

Robert Floden, Michigan State University 

Jeannie Kleinhammer-Tramill, University of South Florida 

Mike Puma, Chesapeake Research Associates 

Paul Sindelar, University of Florida 


Technical Working Group Members for the Evaluation Contract (ED-04-CO-0059/0Q22) 

Stephanie A1 Otaiba, Florida State University 

Dan Black, University of Chicago 

Vincent Connelly, University of New Hampshire 

Batya Elbaum, University of Miami 

Robert Floden, Michigan State University 

Guido Imbens, Harvard University 

Marleen Pugach, University of Wisconsin-Milwaukee 

Daniel Reschly, Vanderbilt University 

Jeff Smith, University of Michigan 

Deborah Speece, University of Maryland 
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